Support

Expand all | Collapse all

Test Case Status Across Maintenance Releases

  • 1.  Test Case Status Across Maintenance Releases

    Posted 27 days ago
    Hello,
    We need to show test case results across releases that show the latest results and not based on Jama priority Here is our use case:
    We have executed all our test cases release 1.0 and now we need to run tests for release 1.1. We do not rerun all our test cases, thus we need to be able to show test results from release 1.0 with the test case results from release 1.1. If a test case failed in release 1.0 which was fixed in release 1.1 we do not get the correct results. We need to get latest run.

    • We cannot archive release 1.0 as these results are needed coverage report
    • We cannot create the maintenance release in the same test plan as this gets to be a real pain trying to pick out the correct test cases to include in a cycle. We need to choose specific test cases based on previous release results and specific items for regression. If we start moving everything around into new groups in the same test plan for release 1.0 and 1.1, this makes it hard to see the status of the test plan for release 1.0.

    What is Jama's solution for handling the above scenario? It would seem like a pretty common use case for regulated development teams, no?

    Regards
    Jason


    ------------------------------
    Jason Barton
    VMS
    Palo Alto CA
    6504246216
    ------------------------------


  • 2.  RE: Test Case Status Across Maintenance Releases

    Posted 26 days ago

    Hi Jason,

    Thank you for reaching out. I've reached out to Consulting team regarding your use case/request. We are not certain there is a way to accommodate this specific request.

    However, it's possible that a custom report could meet your need. With a custom report, you could identify the latest Run across Plans, regardless of Status priority, instead of referring to the Test Case Status in the UI.

    Though by doing this, you could run the chance of confusing the users as the UI and report may show Test Case Statuses differently.

    Please let me know if you have any questions.



    ------------------------------
    Decoteau Wilkerson
    Jama Software
    OR
    ------------------------------



  • 3.  RE: Test Case Status Across Maintenance Releases

    Posted 26 days ago

    Decoteau,

    Thank you for your prompt response. I am curious if there is a different process we should be following? What do you recommend for teams that need to create a maintenance release and wish to run regression tests on a subset of the overall test cases for the product? We still need to show full coverage, I would think this is a common use case, no?

     

    Thank you for your assistance.

    Jason

     

     

     

     

    Jason Barton 
    Digital Health Tools and Operations Excellence Manager

    *For engineering applications requests or issues,

    submit a ticket via the new Engineering Portal

     

    Varian Medical Systems
    3100 Hansen Way, Palo Alto, CA 94304 US 

    jason.barton@varian.com

    T +1.650.424.6216 | M +1.415.515.6616
    varian.com


    Varian Logo

     

     

     


    Varian Confidential






  • 4.  RE: Test Case Status Across Maintenance Releases

    Posted 26 days ago

    Hi Jason,

    This seems like a unique use case to me. However, there may be other Orgs/Users running through this same scenario. 

    I've put in a request for guidance/recommendations from our Product team. In the meantime, I am curious whether other Community Members will have any feedback to offer. 

    I'll post here with any additional information I can find! 



    ------------------------------
    Decoteau Wilkerson
    Jama Software
    OR
    ------------------------------



  • 5.  RE: Test Case Status Across Maintenance Releases

    Posted 24 days ago
    Hi all,
    we have a similar scenario, even more "complicated" due to our product line approach.
    We use  context sensitive custom reports which we feed with ad hoc advanced filters.
    To come to the point, we tag certain test runs that we want to be selected by a specific advanced filter. The tagging occurs either per REST API script or manually.
    Best,
    Alessandro

    ------------------------------
    Alessandro
    Systems Engineer
    SICK AG
    ------------------------------



  • 6.  RE: Test Case Status Across Maintenance Releases

    Posted 24 days ago
    Thank you for sharing this, Alessandro!

    Jason & Alessandro - I've run this use case by members of my team and have some suggested workarounds:

    • If are hoping to designate tests for regression testing, you could Tag the existing Test Cases and add those to a new Test Plan for regression testing. Here the distinction of a regression test is in Test Plan (same Test Cases, different Test Plan.)
    • If the need is to distinguish the Test Cases themselves - those that are run as design verification activities vs those that are run as regression tests - and you don't want the regression testing to impact the existing Test Cases, you could create a new Project (or a new Set in the same Project.) Then you could reuse Tests designated for regression testing (which could be tagged as such) into that designated Project or Set.

      • These new, re-used items could then trace to the Requirements and have their own Runs and Test Cases Status from those Runs.

      • A Test Plan called "Regression Tests" houses these Test Cases in the Regression Tests Project. The location of the item designates them as regression tests and they do not impact the original testing.
      • If Relationships to Requirements are made, you would see the Test Cases downstream from the Requirements.

        Please let me know if you have any questions! 


    ------------------------------
    Decoteau Wilkerson
    Jama Software
    OR
    ------------------------------



  • 7.  RE: Test Case Status Across Maintenance Releases

    Posted 24 days ago
    Decoteau & Alessandro
    Thank you for your input, but if I understand Jama correctly the above does not address our reporting issue, correct?
    Again our use case is we have a full test effort in release 1.0 and we have some failed test cases in release 1.0. In the maintenance release 1.1, we re-ran some of the failed test cases and they passed. We then want to show for maintenance release 1.1 that we have full coverage by showing the results from 1.0 and the regression test from 1.1.  If I understand correctly the Test Cases that failed in 1.0 will still show as failed in trace reports even if though they passed in the 1.1 regression testing effort, correct?



    -A new test plan does not resolve this as the priority goes across test plans
    -I am not sure I follow the copy of test cases fully. If I create a copy within the project of the test cases for regression, I can create a filter that would include the test cases from 1.0, 1.1 and then exclude the failed test cases from 1.0 that were tested again in 1.1? Do I have that right?







    ------------------------------
    Jason Barton
    VMS
    Palo Alto CA
    6504246216
    ------------------------------



  • 8.  RE: Test Case Status Across Maintenance Releases

    Posted 21 days ago
    Hi Jason,

    It sounds like you need multiple Test Plans to be Active, but do not want to have the Status "False" prioritization to apply.

    Unless you Archived a Test Plan with Failed Statuses, that Status will still appear in the corresponding Test Case.

    This specific request would be considered a Feature Request about the application of prioritized Statuses.

    However, you could use a custom report that to disregard the Test Case Status field and generate the Passed Test Runs across Plans (and potentially include corresponding Release information.)



    ------------------------------
    Decoteau Wilkerson
    Jama Software
    OR
    ------------------------------



  • 9.  RE: Test Case Status Across Maintenance Releases

    Posted 21 days ago
    That's exactly my approach. The Test Case Status is not helping in our case and we use a custom report.

    ------------------------------
    Alessandro
    Systems Engineer
    SICK AG
    ------------------------------



  • 10.  RE: Test Case Status Across Maintenance Releases

    Posted 19 days ago
    Looping back to share this new Ideation Card - It looks like this user is suggesting a very similar request to our conversation here: Test Plans per Release - Ideation Card

    ------------------------------
    Decoteau Wilkerson
    Jama Software
    OR
    ------------------------------



  • 11.  RE: Test Case Status Across Maintenance Releases

    Posted 19 days ago

    I created  the idea to have test run results based on most recent run

     

     

    Jason Barton 
    Digital Health Tools and Operations Excellence Manager

    *For engineering applications requests or issues,

    submit a ticket via the new Engineering Portal

     

    Varian Medical Systems
    3100 Hansen Way, Palo Alto, CA 94304 US 

    jason.barton@varian.com

    T +1.650.424.6216 | M +1.415.515.6616
    varian.com


    Varian Logo

     

     

     


    Varian Confidential






  • 12.  RE: Test Case Status Across Maintenance Releases

    Posted 19 days ago
    Good point Decoteau,

    we can go a step further, by adding the field release to the item types
    • test plan,
    • test cycle and
    • test runs.
    This is not enabled in the UI.

    Best,
    Alessandro

    ------------------------------
    Alessandro
    Systems Engineer
    SICK AG
    ------------------------------



  • 13.  RE: Test Case Status Across Maintenance Releases

    Posted 19 days ago
    Great! Thank you for sharing!

    ------------------------------
    Decoteau Wilkerson
    Jama Software
    OR
    ------------------------------