Product Idea

Expand all | Collapse all

Completely lock Test Plans and Test Runs

  • 1.  Completely lock Test Plans and Test Runs

    Posted 08-01-2018 15:37
    Hi -
    We would like to completely lock Test Plans and Test Runs they way Test Cases can be locked using the Workflow feature (on 8.25). We are a medical device company and need to tightly control ALL of our test artifacts, not just Test Cases.

    I plan to use a combination of baselines and exported data to achieve the version control we need, but it would be really nice to have the additional support of the Workflow feature.


    Fujifilm SonoSite

  • 2.  RE: Completely lock Test Plans and Test Runs

    Posted 08-08-2018 15:35

    We recently did a large research project around Verification and Validation and this topic made the list.

    We have already addressed the top 3 findings which included the review of test plans and test results, test runs included in the trace view, and an enhancement to licensing which addresses gaps around the collaborator license.

    The next incremental improvements I have identified are focused on importing or test results, more efficient test result entry and workflow.

    My ideas for workflow is to focus on the test cycle. Possibly 3 states (Complete, Active, Planning) with the test results being locked when the test cycle is in a Complete or Planning state. Does this sound like something that would fit you scenario? Is it a higher priority than say importing test results via Excel?

    Steve Gotsch
    Product Manager
    Jama Software

  • 3.  RE: Completely lock Test Plans and Test Runs

    Posted 08-09-2018 15:10
    Hi Steve -
    Thank you for responding to my post. I apologize for posting a duplicate Feature Request. I read your post on the thread "Test Management Roadmap" shortly after it was published. I guess I forgot that Test Run workflow had already been identified for your upcoming efforts. My bad.

    Regarding managing Test Run workflow through the Test Cycle - I'm not sure about that. My first reaction is that it's a reasonable implementation for users that like the Test Group/Test Cycle structure. Since the Test Group/Test Cycle structure doesn't work well for my organization, it seems like it's just adding unnecessary complexity. I am still learning Jama, but at this point I would prefer to be able to create a Workflow for Test Runs (and Test Plans, for that matter) on the Admin page.

    If the Test Cycle had 3 states, we would probably want the Test Runs to remain unlocked during the Planning state so the test lead could edit the configuration of the Test Runs (implemented as relationships to custom Configuration item types or custom fields). We wouldn't want the Test Run to be available for executing, though.

    I would like to share our experience with the Test Group/Test Cycle structure. So far it's a bit awkward to implement our process using it. It is definitely helpful to be able to create batches of Test Runs simultaneously, but we might need more control over some of the details. Let me explain.
    Here's a brief description of our current process.

    1. We identify the requirements impacted by a set of changes (bug fixes, new features, etc). For a new product the analysis is simple as we plan to verify all the requirements.
    2. We identify the test cases associated with each requirement.
    3. We plan to execute each test case at least once (multiple times if there are multiple configurations/variations, which results in multiple "Test Runs" of a Test Case per software build.) and document the configuration necessary for each test run.
    4. We always find bugs in the first candidate build, and so we need to plan a second (and third, and fourth, unfortunately) round of testing.
    Questions and Challenges
      - Sometimes we halt testing on the first build before all the Test Runs have been executed. What do we do with the Test Runs that haven't been executed? We can't move them from one Test Cycle to another (or remove entirely), and we can't configure the Test Run Status field to add an option that describes this situation (like "Abandoned"). It just looks like there's work left undone. I know there are ways around this, but it would be nice if Jama handled the situation more gracefully.
      - When planning for the second build, we have to evaluate the changes since the first build and choose the set of Test Cases to run accordingly. I know we can use the status of the previously executed Test Runs to determine which Test Runs to create, but sometimes it's not that simple. Depending on the change we might need to execute tests that passed or failed or weren't run at all. We could have all 3 scenarios in the same Test Group - it's impossible to determine in advance. What's the best way to select a new set of Test Cases to create Test Runs? I know we can move Test Cases between Test Groups after Test Cycles have been created, and that seems like the best option so far.

    Regarding the priority of Test Run workflow and importing test results via Excel - I would say they are equal. Different people will use each feature and to them one will be more important than the other, but to the team as a whole I think they are equally important.

    Thank you for reading my lengthy response. I hope it provided some useful, if not straightforward, input. Let me know if you have any other questions about this.

    All three of the areas you identified for incremental improvements will be a big help to us. Do you have any idea regarding the timeframe for these efforts? Are you planning to post updates on the Jama community? I have seen some Product Bulletins (the most recent almost a year old) and I just found the Announcements page, so I'll watch those 2 areas. Are there any other places I should look?

    Please excuse my eagerness for information. I am responsible for planning and rolling out Jama to our test teams over the next few months and I want Jama to be received well.


    Theresa Soares
    Verification and Validation Engineer III, System Verification
    FUJIFILM SonoSite, Inc.

    425.442.1523 mobile

    NOTICE:  This message, including any attachments, is only for the use of the intended recipient(s) and may contain confidential, sensitive and/or privileged information, or information otherwise prohibited from dissemination or disclosure by law or regulation, including applicable export regulations.  If the reader of this message is not the intended recipient, you are hereby notified that any use, disclosure, copying, dissemination or distribution of this message or any of its attachments is strictly prohibited.  If you received this message in error, please contact the sender immediately by reply email and destroy this message, including all attachments, and any copies thereof.

  • 4.  RE: Completely lock Test Plans and Test Runs

    Posted 08-17-2018 16:23
    Great response - you gave me a lot to think about.

    I do not have a timeline for any of these, of course, but we will be continuing our research and will be looking for more feedback ahead of moving forward on any development. As you've outed yourself and interests to me we'll let you know what we are thinking.

    Thanks again, Steve

    Steve Gotsch
    Product Manager
    Jama Software