View Only
Expand all | Collapse all

Many Variations on a Test Case

  • 1.  Many Variations on a Test Case

    Posted 03-17-2015 11:28
    I'm wondering if anyone has a "trick" for running a single test case many times within a test plan?

    We're facing a kind of one-off project where we have to verify something like 60-80 different configurations of, essentially the same device (our team tests embedded firmware).  The closest analogy I can think of is browser testing where one might want to run the same test case against Firefox, Chrome, etc. but the numbers is are big by comparison.

    I think most of us at ISC have tended to use Cycles to do this, but usually for a comparatively small number, 3, 4 or 5 sort of things.  Creating 60-80 Cycles seems pretty daunting...

    Any ideas?


  • 2.  Re: Many Variations on a Test Case

    Posted 03-17-2015 11:49
    Why wouldn't you have one test case for each browser?  Whether a test case passes or fails, could be dependent on each individual browser.  It seems you would want to capture that at the browser level.  That is how we would set that up here.  So instead of creating 60-80 cycles for one test case, you would have 60-80 test cases with one cycle.

  • 3.  Re: Many Variations on a Test Case

    Posted 03-17-2015 13:39
    I would do that with different testplans, at least in the browser case. But 60-80 configs would be a lot of testplans, but it has the nice effect, only if all testruns for a testcase are passed the testcase status goes to passed.

  • 4.  Re: Many Variations on a Test Case

    Posted 03-17-2015 13:49
    Hey Angela-

    For browser compatibility testing I think that might be a solution with reuse ... though I've not used it yet other than to copy a test case.

    I offered the browser example as something familiar, and we know some test management tools allow one to create multiple Test Runs from a single test cases given one or more lists of variants, which I expect some in our community are familiar with.  It's a pretty common feature.

    My task is very likely to be a one-time thing where we want to run 2 test cases; one against the entire batch of 60-80 samples, and the other against some representatives of various groups across the entire set, maybe 10-15%.   I'm trying avoid a lot of work for something I'll do once.

    I hate the thought, but a colleague and I discussed this and he suggested what I had in the back of my head ... Just use Excel and attach it to the Test Plan for archival.


  • 5.  Re: Many Variations on a Test Case

    Posted 03-17-2015 14:16
    Hey Harald-

    I think I did that early on (for 2-3 variants) but we felt the reporting was confusing (at least).  Few people could get comfortable with why multiple test plans., and we have just 1 paper test plan.   I don't recall what the issue was with reporting, but it certainly wasn't an intuitive use of Test Plans.

    I think multiple Cycles would actually be somewhat less work, but still quite a bit.  For that the group and test plan could be shared.  Jama, reasonably, treats Cycles as ordered, and I found that confused some folks a little (when the testing really wasn't ordered)...

  • 6.  Re: Many Variations on a Test Case

    Posted 03-17-2015 14:35
    Hi there. I don't think this sheds any drastic new light on the subject but I once put together a PowerPoint deck for a Jama customer on this very topic. It's attached with 4 basic options for how to handle TC Variations.  You can download/view it here:

    Let me know if this helps at all.

  • 7.  RE: Re: Many Variations on a Test Case

    Posted 12-12-2017 17:38
    I just tried accessing the slide deck that Preston linked to, but unfortunately it appears the link is dead.  Can anyone provide a copy of the slide deck or can Jama re-post it?


    Plexus Corp (On-Premise)

  • 8.  RE: Re: Many Variations on a Test Case

    Posted 12-13-2017 14:27
      |   view attached
    Hello Trent, 

    I have downloaded the presentation for you; I have attached it here.

    Getro Naissance
    Jama Software


  • 9.  Re: Many Variations on a Test Case

    Posted 03-17-2015 14:53
    It's a good summary of some options we discussed and tinkered with.  The last page is particularly helpful.

    One thought that came to mind as I read through raises another question that came up...  We've wondered what happens to Test Runs in a completed Cycle if the referenced Test Cases are deleted later?

    I ask because I'm wondering about using an import to quickly create 60-80 test cases** ... and then deleting them after the Cycle is complete Since we'll be very unlikely to use them again.   (Each item to test will have a unique part number and I could do some Excel stuff to add this to the Name and Description of each copy of the original Test Case.)


    ** Something I read in the slides triggered the idea-    ;-)

  • 10.  Re: Many Variations on a Test Case

    Posted 03-17-2015 14:56
    Test Runs will still remain in the cycle, even if the Test Cases are deleted later. Jama treats them as independent items in the database.  The downside is that any defects logged from these Test Runs wouldn't have upstream traceability to requirements (via the TC) given that you've deleted the Test Cases and/or never related them to the Reqs.

  • 11.  Re: Many Variations on a Test Case

    Posted 03-17-2015 17:00
    The upstream Traces to the testruns should remain.