Test runs, cycles and plans should be able to be locked down so no changes can be made

Shankar Subramanian
edited July 2016 in
Looking for a way to lock the "Test Run"/"Test Cycle" in Jama test management system, so that the state of the Test Run can be frozen at a point of time for records.
When we lock the Test Run, the system should not allow any more changes or updates to the test cases associated with the test run.
  • This will help in test results being logged in a wrong or  incorrect Test Run even by accident or mistake.
  • This will help us in always going back and able to check the state of the test run when it was complete [Test Run will be locked at the end of the cycle/time].

Comments

  • Kristina King
    edited June 2016
    Shankar, thanks for joining the community and asking this question.
    The Test Case resides at the very top, so to speak. Although any test run derived from it can affect the Test Case's status (how depends on the priority of the executed status), edits or locks made to Test Runs are not designed to affect the Test Case. 

    As far as issue "this will help in test results being logged in a wrong or  incorrect Test Run even by accident or mistake," I don't have a way around preventing user error.

    As far as "This will help us in always going back and able to check the state of the test run when it was complete [Test Run will be locked at the end of the cycle/time]", goes I do have some follow-up questions.
    -Do you archive your Test Plans after the cycle is complete? Archiving a Test Plan ensures that new cycles are not created for that particular plan and maintains a history of the "state of the test run." (It may mean you would have to create less test cycles per plan and archive more frequently, though, depending on your current process.) However, this still means the Test Case status will reflect the most recent/highest priority status.
    -This is by design, though—is there a reason you don't want your Test Cases to reflect these predetermined statuses?
    -If it's because of product variances (versions or releases), you might consider using Reuse to create multiple copies of the same test case, and put the pertinent cases in different test plans (with very clear naming conventions!). Because Test Case Status is not an editable field, but rather pre-calculated, you can never really lock it, so it might end up being a matter of training and process. 
  • Hi Kristina,
    Thanks for the response. Please find my responses and a question where your suggestion how Jama's Test Management can be used will be useful.

    -Do you archive your Test Plans after the cycle is complete?
    >> At present our practice is to generate a master test suite for the project and run iterations of this test plan. But we can make a process here where we create a test plan for every release based on the release and archive the test plan once the execution is complete.

    -This is by design, though—is there a reason you don't want your Test Cases to reflect these predetermined statuses?
    >> Like I said earlier Jama should be working on the idea of a test plan getting executed once and another test plan created based on the requirement where our org is working in the style of generating one test plan for the project and execute the in iterations during the releases.
    So here you have an option of creating a fresh test run/cycle [test cases will have no results marked] or create a followup test run/cycle where the previously executed test case results will be predetermined.

    -If it's because of product variances (versions or releases), you might consider using Reuse to create multiple copies of the same test case, and put the pertinent cases in different test plans......
    >> I agree and understand on reusing the test plans or creating new test plan according to the release scope and am working on the style to define as a process.
    But in this regard I have a scenario and would like your suggestion or how to go about it.
    In our org we have a Regression Test Suite/Daily Sanity Test Plan which is executed on a daily basis in order catch regressions at the earliest. In order to achieve this we create a test plan for regressions and create test cycles/runs in the test plan on a daily basis that will be populated with the test results.
    With the Jama Test Management, how do we about achieving this practice? One idea is that keep creating/reusing the test plan on a daily basis. Please suggest if there is a better way of implementing this process.
  • Kristina King
    edited June 2016
    In our org we have a Regression Test Suite/Daily Sanity Test Plan which is executed on a daily basis in order catch regressions at the earliest. In order to achieve this we create a test plan for regressions and create test cycles/runs in the test plan on a daily basis that will be populated with the test results.With the Jama Test Management, how do we about achieving this practice? One idea is that keep creating/reusing the test plan on a daily basis. Please suggest if there is a better way of implementing this process.

    Thanks for the clarification. There is a bit of a tweak I suggest to this process...
    If you're reusing only the test plan for each daily regression cycle, your test case status is going to reflect the most recent/highest priority test case, so you will not have the intended "freeze" effect on the Test Case itself. This is no different than just creating a new Test Cycle for each regression. (And creating a new cycle is certainly the lightweight option, so I'd recommend that.) But it doesn't sound like that's what you want. If you want to create a record where the Test Case itself is frozen in time with an immutable status, you'd have to first reuse the Test Cases for the plan, and then create a new test plan. (Reusing the test plan would cause updates to those Test Case Statuses that you want to keep frozen.) This is a bit of a lift.
    But this extra work is unavoidable if you absolutely have to freeze the Test Case Status, just given the way Jama was designed to work. 
  • Shankar Subramanian
    edited February 2016
    Hi Kritina,
    Thanks for the response.
    I tried to duplicate a test plan and I get a brand new plan with the test cases having no TestCaseStatus updated. So instead of re-using a test plan on daily basis, I can create a duplicate copy of the test plan and archive the same on daily basis before creating the next duplicate.
    The flip side I see is the number of test plans that the project is gong to have.

    Request you to take this as a feature request which when enabled will allow you to freeze a test run of a test plan [to retain the set of TestCaseStatus] and create a new one for a fresh execution of the test plan.
    This will enable you to get the history of the test case execution and its TestCaseStatus and associated data [e.g. the build used] during various execution cycles and the data will be very useful to understand the trend and find whether the regression is recurring regression and so on.

    Thanks,
    Shankar Subramanian
  • Kristina King
    edited June 2016
    Hi Shankar,
    I'll update this thread to an "Idea" so that we consider this feature request.
  • Shankar Subramanian
    edited February 2016
    Thank you Kristina. Looking forward to share more ideas.

    Thanks,
    Shankar Subramanian
  • George Edwards
    George Edwards SAFe Release Train Engineer, BioContinuum Process Automation & Delivery
    edited June 2016
    This reply was created from a merged topic originally titled Idea: Limit user ability to modify or delete test cycles.

    We had a recent report where somehow a tester deleted a test cycle. At around the same time a PM from a separate project asked about how to lock down test results to prevent any subsequent changes being made to them. I contacted Jama SupportPaul Tseng about these. He indicated that a test cycle can't be restored nor can test results be locked down. He sounded a little surprised that noone had flagged the issue before now.

    I mention the two points together because the deleted test cycle is an example of what the PM would like to prevent from happening. A variant of this concern is preventing folks from subsequently going in and changing test results.

    A test plan can certainly be archived but as I understand it that doesn't actually prevent its test cycles from being altered.

    Unless there's a reason to keep test cycles always editable, I'd like to propose that Jama functionality be tweaked to allow PMs to lock down a test plan and/or test cycle to prevent it from being altered, and certainly from being deleted. Or, perhaps access to the functionality should be limited to PMs. Is that something that could be supported in the future?

    At another level, the UI is a bit misleading. Those green circles with a plus inside (see Releases, Baselines, Test Plan tabs) usually indicate functionality that only the PM can see. However, with Test Plans, any user with write permissions can create a test plan (and add and modify test cycles). This seems inconsistent.
  • Kristina King
    edited June 2016
    Hi George! I just merged your Idea with an existing thread in order to consolidate the conversation (and the votes). I don't think you're the first to inquire about locking down test artifacts (cycles/runs/plans), but I don't think it's something we hear much about in tickets.

    I think you raise good points—I don't have anything to say in terms of a functional rebuttal! I can comment on the last part about the UI, though—originally Project Manage permissions were required for the editing or creation of test plans. After much user feedback we opted to require project Create permissions instead in the June 21, 2014 hosted release. However, we did not change the UI to reflect the change, so you're right, it does feel inconsistent. However, it was intentional.
  • Trent Hirsch
    edited May 2018
    Hi @Kristina,

    I'd like to up-vote the request to make it possible for test cycles to be locked down.  As described in another thread (How to prevent test cycle from being deleted?), the ability to delete test cycles even when the parent test plan and the child test runs are locked is dangerous and inconsistent with Jama's functionality elsewhere.  For instance, if a user attempts to delete a folder of test cases where any of the test cases are locked, the locked test cases are spared.  As well, if a user attempts to "Make Test Runs Current" on a test cycle where any of the test runs that would otherwise be updated are locked, the locked test runs are spared.

    Control of test records is a big deal when executing verification testing in regulated industries.  Considering that a) there is no warning message for the user when attempting to delete a test cycle like there is for "Make Test Runs Current" and b) test runs cannot be recovered if deleted, this is a serious vulnerability for our company. 

    Has there been any progress on getting this feature into the backlog?

    Thanks,

    ------------------------------
    Trent
    Plexus Corp (On-Premise)
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 03-22-2016 15:28
    From: Kristina King
    Subject: Re: Test runs, cycles and plans should be able to be locked down so no changes can be made

    Hi George! I just merged your Idea with an existing thread in order to consolidate the conversation (and the votes). I don't think you're the first to inquire about locking down test artifacts (cycles/runs/plans), but I don't think it's something we hear much about in tickets.

    I think you raise good points-I don't have anything to say in terms of a functional rebuttal! I can comment on the last part about the UI, though-originally Project Manage permissions were required for the editing or creation of test plans. After much user feedback we opted to require project Create permissions instead in the June 21, 2014 hosted release. However, we did not change the UI to reflect the change, so you're right, it does feel inconsistent. However, it was intentional.
  • Kristina King
    edited May 2018
    Hi @Trent,

    I can make no sort of promise, as the roadmap can always change based on security vulnerabilities (Meltdown and Spectre), new regulations (GDPR), or a pivot in product strategy. But this need is definitely something we're aware of; we recently made some changes to the Test Center (now known as Verification & Validation) and plan for that work to continue. I checked in with our Director of Product and he informed me the ability to lock down results or testing workflows is at the top of mind! So I'm cautiously optimistic that we will see this update.
  • Sumit Benarji
    edited May 2019
    Hi Kristina,

    Do we have this feature available in JAMA 8.25 wherein I can lock test cycle once test cycle is completed ? Purpose is to preserve/make the status of test  cases read only and also to prevent someone deleting test cycle.
  • Kristina King
    edited May 2019
    Hi Sumit,

    The ability to lock a test cycle has not been added to Jama Connect.
  • Tom Fuller
    edited November 2019
    Please consider adding permissions to lock test cycles and create new test cycles.  We have had testers execute test runs on old (abandoned) cycles (due to a new release candidate), and waste precious time at the end of a project.  Then they need to re-run those tests on the new cycle.

    Also, we have people adding new test cycles (with odd names) when they should not.  This causes wasted time.

    User Story 1: As a Manager, I need to be able to control when test cycles can be run, so that the proper test runs are executed.  Test cycles should have the ability (controlled by user permissions) to be turned on and off, with a history of who made changes to these states, and when.
    User Story 2: As a Manager, I need to be able to control who can create test cycles, so that unnecessary cycles are not created.


    -------------------------------------------
    Original Message:
    Sent: 05-14-2019 14:23
    From: Kristina King
    Subject: Re: Test runs, cycles and plans should be able to be locked down so no changes can be made

    Hi Sumit,

    The ability to lock a test cycle has not been added to Jama Connect.

    ------------------------------
    Kristina King Manager, Technical Support
    Jama Software
    ------------------------------

    Original Message:
    Sent: 05-14-2019 08:20
    From: Sumit Benarji
    Subject: Re: Test runs, cycles and plans should be able to be locked down so no changes can be made

    Hi Kristina,

    Do we have this feature available in JAMA 8.25 wherein I can lock test cycle once test cycle is completed ? Purpose is to preserve/make the status of test  cases read only and also to prevent someone deleting test cycle.

    ------------------------------
    Sumit Benarji

    Original Message:
    Sent: 05-21-2018 16:08
    From: Kristina King
    Subject: Re: Test runs, cycles and plans should be able to be locked down so no changes can be made

    Hi @Trent,

    I can make no sort of promise, as the roadmap can always change based on security vulnerabilities (Meltdown and Spectre), new regulations (GDPR), or a pivot in product strategy. But this need is definitely something we're aware of; we recently made some changes to the Test Center (now known as Verification & Validation) and plan for that work to continue. I checked in with our Director of Product and he informed me the ability to lock down results or testing workflows is at the top of mind! So I'm cautiously optimistic that we will see this update.

    ------------------------------
    Kristina King
    Jama Software

    Original Message:
    Sent: 05-18-2018 18:26
    From: Trent
    Subject: Re: Test runs, cycles and plans should be able to be locked down so no changes can be made

    Hi @Kristina,

    I'd like to up-vote the request to make it possible for test cycles to be locked down.  As described in another thread (How to prevent test cycle from being deleted?), the ability to delete test cycles even when the parent test plan and the child test runs are locked is dangerous and inconsistent with Jama's functionality elsewhere.  For instance, if a user attempts to delete a folder of test cases where any of the test cases are locked, the locked test cases are spared.  As well, if a user attempts to "Make Test Runs Current" on a test cycle where any of the test runs that would otherwise be updated are locked, the locked test runs are spared.

    Control of test records is a big deal when executing verification testing in regulated industries.  Considering that a) there is no warning message for the user when attempting to delete a test cycle like there is for "Make Test Runs Current" and b) test runs cannot be recovered if deleted, this is a serious vulnerability for our company.

    Has there been any progress on getting this feature into the backlog?

    Thanks,

    ------------------------------
    Trent
    Plexus Corp (On-Premise)

    Original Message:
    Sent: 03-22-2016 15:28
    From: Kristina King
    Subject: Re: Test runs, cycles and plans should be able to be locked down so no changes can be made

    Hi George! I just merged your Idea with an existing thread in order to consolidate the conversation (and the votes). I don't think you're the first to inquire about locking down test artifacts (cycles/runs/plans), but I don't think it's something we hear much about in tickets.

    I think you raise good points-I don't have anything to say in terms of a functional rebuttal! I can comment on the last part about the UI, though-originally Project Manage permissions were required for the editing or creation of test plans. After much user feedback we opted to require project Create permissions instead in the June 21, 2014 hosted release. However, we did not change the UI to reflect the change, so you're right, it does feel inconsistent. However, it was intentional.