Jama Software Automatically Passes Skipped Test Steps

[Deleted User]
[Deleted User] Member Posts: 17
For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama Software will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama Software to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

------------------------------
Ken G
ASR
------------------------------

Comments

  • [Deleted User]
    [Deleted User] Member Posts: 21
    edited June 2017
    We have found the same issue. We don't always run the tests in the same sequence (equipment availability) and JAMA Software erroneously marks skipped tests as pass. JAMA Software has refused to fix the problem.

    ------------------------------
    Randy - Delta Controls
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 04-27-2017 01:11
    From: Ken G
    Subject: Jama Software Automatically Passes Skipped Test Steps

    For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama Software will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama Software to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

    ------------------------------
    Ken G
    ASR
    ------------------------------



  • [Deleted User]
    [Deleted User] Posts: 911
    edited June 2017
    Hi Ken, 
    It is by design that Jama Software passes all steps in this circumstance. For better or worse (in yours and Randy's case, worse), Jama Software is designed to pass the whole test if the final step is passed. The steps function assumes that steps are completed in numerical order. But Randy is right; at this point we don't have any plans to change this.

    ------------------------------
    Kristina King
    Jama Software
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 04-27-2017 01:11
    From: Ken G
    Subject: Jama Software Automatically Passes Skipped Test Steps

    For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama Software will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama Software to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

    ------------------------------
    Ken G
    ASR
    ------------------------------



  • Lien Bäcker
    Lien Bäcker Member, Data Exchange, Jama Connect Interchange™ (JCI) Posts: 83
    edited June 2017
    Hi Kristina,

    I see the Arguments behind this Feature. But for our tester, it was perceived more or less as an issue. It is not always the case that the steps were written, that step 3 passed meant step 1 and 2 should be passed.
    For us, it would be better, when the tester could choose step 3 passed, Jama Software should not do anything with the result of step 1 and 2.

    Thanks

    ------------------------------
    Lien Bäcker
    Gira, Giersiepen GmbH & Co. KG
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 05-05-2017 19:07
    From: Kristina King
    Subject: Jama Automatically Passes Skipped Test Steps

    Hi Ken, 
    It is by design that Jama Software passes all steps in this circumstance. For better or worse (in yours and Randy's case, worse), Jama Software is designed to pass the whole test if the final step is passed. The steps function assumes that steps are completed in numerical order. But Randy is right; at this point we don't have any plans to change this.

    ------------------------------
    Kristina King
    Jama Software
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 04-27-2017 01:11
    From: Ken G
    Subject: Jama Automatically Passes Skipped Test Steps

    For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama Software will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama Software to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

    ------------------------------
    Ken G
    ASR
    ------------------------------








  • [Deleted User]
    [Deleted User] Posts: 3
    edited June 2017
    I can imagine that this functionality to auto pass is useful when reviewing requirements, but for testing, this is risky. I can't imagine a tester reviewed this functionality when it was implemented and said 'yes let's auto skip steps by default to save time.
    It takes away from the detailed approach of testing and not missing items.

    There are a few ways things function in Jama Software that doesn't make sense for testing. I understand people can approach testing differently, but in testing there are a few obvious things that are mandatory for testing (not skipping steps, providing more than just Passed or Failed for a test step - i.e descoped, or N/A)


    ------------------------------
    Diana
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 05-05-2017 19:07
    From: Kristina King
    Subject: Jama Automatically Passes Skipped Test Steps

    Hi Ken, 
    It is by design that Jama passes all steps in this circumstance. For better or worse (in yours and Randy's case, worse), Jama is designed to pass the whole test if the final step is passed. The steps function assumes that steps are completed in numerical order. But Randy is right; at this point we don't have any plans to change this.

    ------------------------------
    Kristina King
    Jama Software
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 04-27-2017 01:11
    From: Ken G
    Subject: Jama Automatically Passes Skipped Test Steps

    For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

    ------------------------------
    Ken G
    ASR
    ------------------------------








  • [Deleted User]
    [Deleted User] Posts: 911
    edited June 2017
    Hi Diana,

    This feature was specifically designed to function this way, and like all features, there was real-world testing before it shipped. But this was many years ago, and the average Jama Software customer might have changed since then. I'm curious-is your company in the same position as Ken's, where Step order does not necessarily mean Execution order?

    ------------------------------
    Kristina King
    Jama Software
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 06-22-2017 21:29
    From: Diana Matic
    Subject: Jama Automatically Passes Skipped Test Steps

    I can imagine that this functionality to auto pass is useful when reviewing requirements, but for testing, this is risky. I can't imagine a tester reviewed this functionality when it was implemented and said 'yes let's auto skip steps by default to save time.
    It takes away from the detailed approach of testing and not missing items.

    There are a few ways things function in Jama that doesn't make sense for testing. I understand people can approach testing differently, but in testing there are a few obvious things that are mandatory for testing (not skipping steps, providing more than just Passed or Failed for a test step - i.e descoped, or N/A)


    ------------------------------
    Diana
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 05-05-2017 19:07
    From: Kristina King
    Subject: Jama Automatically Passes Skipped Test Steps

    Hi Ken, 
    It is by design that Jama passes all steps in this circumstance. For better or worse (in yours and Randy's case, worse), Jama is designed to pass the whole test if the final step is passed. The steps function assumes that steps are completed in numerical order. But Randy is right; at this point we don't have any plans to change this.

    ------------------------------
    Kristina King
    Jama Software
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 04-27-2017 01:11
    From: Ken G
    Subject: Jama Automatically Passes Skipped Test Steps

    For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

    ------------------------------
    Ken G
    ASR
    ------------------------------













  • [Deleted User]
    [Deleted User] Posts: 3
    edited June 2017
    Hi Kristina, 
    Step order does mean Execution order generally, but there is a scenario where for example I may need to check that multiple fields have a certain property. In this case, the test steps would have:
    Test Step
    1             Verify that x applies for the following fields
    2              Field 1
    3              Field 2
    4              Field 3
    5              ..etc

    In this case I might verify Field 3 first because I am waiting for something for Field 2. I'd rather split out each field so they each get proper attention than bunching it all into one test case. This is a simple example, but you can see one case where this is applicable.
    Something to consider in a future release.

    Regards, 
    Diana
  • [Deleted User]
    [Deleted User] Member Posts: 17
    edited June 2017
    I would suggest Jama Software to provide an option to either toggle the skip feature to auto pass the skipped steps or not. Default = not.

    ------------------------------
    Thanks,
    Ken G
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 06-25-2017 19:31
    From: Diana Matic
    Subject: Jama Automatically Passes Skipped Test Steps

    Hi Kristina, 
    Step order does mean Execution order generally, but there is a scenario where for example I may need to check that multiple fields have a certain property. In this case, the test steps would have:
    Test Step
    1             Verify that x applies for the following fields
    2              Field 1
    3              Field 2
    4              Field 3
    5              ..etc

    In this case I might verify Field 3 first because I am waiting for something for Field 2. I'd rather split out each field so they each get proper attention than bunching it all into one test case. This is a simple example, but you can see one case where this is applicable.
    Something to consider in a future release.

    Regards, 
    Diana

    ------------------------------
    Diana
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 06-23-2017 11:38
    From: Kristina King
    Subject: Jama Automatically Passes Skipped Test Steps

    Hi Diana,

    This feature was specifically designed to function this way, and like all features, there was real-world testing before it shipped. But this was many years ago, and the average Jama customer might have changed since then. I'm curious-is your company in the same position as Ken's, where Step order does not necessarily mean Execution order?

    ------------------------------
    Kristina King
    Jama Software
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 06-22-2017 21:29
    From: Diana Matic
    Subject: Jama Automatically Passes Skipped Test Steps

    I can imagine that this functionality to auto pass is useful when reviewing requirements, but for testing, this is risky. I can't imagine a tester reviewed this functionality when it was implemented and said 'yes let's auto skip steps by default to save time.
    It takes away from the detailed approach of testing and not missing items.

    There are a few ways things function in Jama that doesn't make sense for testing. I understand people can approach testing differently, but in testing there are a few obvious things that are mandatory for testing (not skipping steps, providing more than just Passed or Failed for a test step - i.e descoped, or N/A)


    ------------------------------
    Diana
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 05-05-2017 19:07
    From: Kristina King
    Subject: Jama Automatically Passes Skipped Test Steps

    Hi Ken, 
    It is by design that Jama passes all steps in this circumstance. For better or worse (in yours and Randy's case, worse), Jama is designed to pass the whole test if the final step is passed. The steps function assumes that steps are completed in numerical order. But Randy is right; at this point we don't have any plans to change this.

    ------------------------------
    Kristina King
    Jama Software
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 04-27-2017 01:11
    From: Ken G
    Subject: Jama Automatically Passes Skipped Test Steps

    For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

    ------------------------------
    Ken G
    ASR
    ------------------------------























  • [Deleted User]
    [Deleted User] Posts: 78
    edited February 2020
    Hi,
    any update?
    We find this behavior very problematic and I expect it to be the same for most users that are not in the Software industry.

    Feature request> Test Run: option to disable Steps auto-fill

    Steps in Test Runs were obviously develop for Software evaluation. Also by the name, it's clear that a sequence of test "steps" is assumed: when defining a step as PASSED or FAILED there is a feature that auto-fills previous steps as PASSED when undefined (NOT_RUN). This is great for clicks saving provided that steps are in sequence.
    For Hardware test, I don't see the need to check steps of a sequence but to check the same Test Case under different external conditions (temperature, loads, supply voltages…). So a use Steps for my different condition but I'm then very annoyed by the auto-fill feature since the conditions were not necessarily defined in the order they are finally tested.
    My proposal is then to add an option in the Test Case configuration to enabling/disabling the step auto fill. It would also be great to be able to customize the Step field and related table headers.
  • Ashley Ernst
    Ashley Ernst Jama Staff, Product Manager Moderator Posts: 45
    edited November 2023

    Hello, thank you all for your ongoing feedback. In an effort to improve testing efficiency and outcomes, we've introduced new options in 9.5 which are relevant to this discussion:

    Organization Administrators may configure how the system will behave when users skip test steps and don't assign them a status:

    • Assign "Passed" to all skipped steps

    • Do not assign status to any skipped steps

    • Allow tester to select status for skipped step (i.e. ignore, pass, fail)

    Instructions:

    1. Navigate to Admin > Organization Admin > Item types
    2. Select the Test Run item type in the left panel
    3. Select the Status field in the right panel and click the pencil icon to edit
    4. From the popup window, select the desired default behavior for completing steps out of order 
    5. Click save

    Organization Administrators may configure "Not applicable"/"N/A" as an additional status option. Once configured, testers will see the option to mark some or all steps in a run as 'Not applicable'. Any steps marked as 'Not applicable' will be excluded from the overall pass/fail result of the run.

    Instructions:

    1. Navigate to Admin > Organization Admin > Item types
    2. Select the Test Run item type in the left panel
    3. Select the Status field in the right panel and click the pencil icon to edit
    4. From the popup window and under the status options, select the box for "Not Applicable (N/A)" status
    5. Click save

    We welcome any additional feedback on these new options, as we're are eager to understand the impact.

    ------------------------------
    Ashley Ernst
    CA
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 02-21-2020 01:39
    From: Christian Binard
    Subject: Jama Software Automatically Passes Skipped Test Steps

    Hi,
    any update?
    We find this behavior very problematic and I expect it to be the same for most users that are not in the Software industry.

    Feature request> Test Run: option to disable Steps auto-fill

    Steps in Test Runs were obviously develop for Software evaluation. Also by the name, it's clear that a sequence of test "steps" is assumed: when defining a step as PASSED or FAILED there is a feature that auto-fills previous steps as PASSED when undefined (NOT_RUN). This is great for clicks saving provided that steps are in sequence.
    For Hardware test, I don't see the need to check steps of a sequence but to check the same Test Case under different external conditions (temperature, loads, supply voltages…). So a use Steps for my different condition but I'm then very annoyed by the auto-fill feature since the conditions were not necessarily defined in the order they are finally tested.
    My proposal is then to add an option in the Test Case configuration to enabling/disabling the step auto fill. It would also be great to be able to customize the Step field and related table headers.

    ------------------------------
    Christian Binard
    ON semiconductor
    Belgium
    ------------------------------

    Original Message:
    Sent: 06-26-2017 02:27
    From: Ken G
    Subject: Jama Software Automatically Passes Skipped Test Steps

    I would suggest Jama Software to provide an option to either toggle the skip feature to auto pass the skipped steps or not. Default = not.

    ------------------------------
    Thanks,
    Ken G

    Original Message:
    Sent: 06-25-2017 19:31
    From: Diana Matic
    Subject: Jama Automatically Passes Skipped Test Steps

    Hi Kristina,
    Step order does mean Execution order generally, but there is a scenario where for example I may need to check that multiple fields have a certain property. In this case, the test steps would have:
    Test Step
    1             Verify that x applies for the following fields
    2              Field 1
    3              Field 2
    4              Field 3
    5              ..etc

    In this case I might verify Field 3 first because I am waiting for something for Field 2. I'd rather split out each field so they each get proper attention than bunching it all into one test case. This is a simple example, but you can see one case where this is applicable.
    Something to consider in a future release.

    Regards,
    Diana

    ------------------------------
    Diana

    Original Message:
    Sent: 06-23-2017 11:38
    From: Kristina King
    Subject: Jama Automatically Passes Skipped Test Steps

    Hi Diana,

    This feature was specifically designed to function this way, and like all features, there was real-world testing before it shipped. But this was many years ago, and the average Jama customer might have changed since then. I'm curious-is your company in the same position as Ken's, where Step order does not necessarily mean Execution order?

    ------------------------------
    Kristina King
    Jama Software

    Original Message:
    Sent: 06-22-2017 21:29
    From: Diana Matic
    Subject: Jama Automatically Passes Skipped Test Steps

    I can imagine that this functionality to auto pass is useful when reviewing requirements, but for testing, this is risky. I can't imagine a tester reviewed this functionality when it was implemented and said 'yes let's auto skip steps by default to save time.
    It takes away from the detailed approach of testing and not missing items.

    There are a few ways things function in Jama that doesn't make sense for testing. I understand people can approach testing differently, but in testing there are a few obvious things that are mandatory for testing (not skipping steps, providing more than just Passed or Failed for a test step - i.e descoped, or N/A)


    ------------------------------
    Diana

    Original Message:
    Sent: 05-05-2017 19:07
    From: Kristina King
    Subject: Jama Automatically Passes Skipped Test Steps

    Hi Ken,
    It is by design that Jama passes all steps in this circumstance. For better or worse (in yours and Randy's case, worse), Jama is designed to pass the whole test if the final step is passed. The steps function assumes that steps are completed in numerical order. But Randy is right; at this point we don't have any plans to change this.

    ------------------------------
    Kristina King
    Jama Software

    Original Message:
    Sent: 04-27-2017 01:11
    From: Ken G
    Subject: Jama Automatically Passes Skipped Test Steps

    For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

    ------------------------------
    Ken G
    ASR
    ------------------------------























    Ashley Ernst
    [Pronouns]
    Product Manager
    Jama Software