Jama Software Automatically Passes Skipped Test Steps

[Deleted User]

For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama Software will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama Software to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

------------------------------
Ken G
ASR
------------------------------

[Deleted User]

We have found the same issue. We don't always run the tests in the same sequence (equipment availability) and JAMA Software erroneously marks skipped tests as pass. JAMA Software has refused to fix the problem.

------------------------------
Randy - Delta Controls
------------------------------
-------------------------------------------
Original Message:
Sent: 04-27-2017 01:11
From: Ken G
Subject: Jama Software Automatically Passes Skipped Test Steps

For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama Software will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama Software to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

------------------------------
Ken G
ASR
------------------------------

[Deleted User]

Hi Ken, 
It is by design that Jama Software passes all steps in this circumstance. For better or worse (in yours and Randy's case, worse), Jama Software is designed to pass the whole test if the final step is passed. The steps function assumes that steps are completed in numerical order. But Randy is right; at this point we don't have any plans to change this.

------------------------------
Kristina King
Jama Software
------------------------------
-------------------------------------------
Original Message:
Sent: 04-27-2017 01:11
From: Ken G
Subject: Jama Software Automatically Passes Skipped Test Steps

For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama Software will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama Software to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

------------------------------
Ken G
ASR
------------------------------

Lien Bäcker

Hi Kristina,

I see the Arguments behind this Feature. But for our tester, it was perceived more or less as an issue. It is not always the case that the steps were written, that step 3 passed meant step 1 and 2 should be passed.
For us, it would be better, when the tester could choose step 3 passed, Jama Software should not do anything with the result of step 1 and 2.

Thanks

------------------------------
Lien Bäcker
Gira, Giersiepen GmbH & Co. KG
------------------------------
-------------------------------------------
Original Message:
Sent: 05-05-2017 19:07
From: Kristina King
Subject: Jama Automatically Passes Skipped Test Steps

Hi Ken, 
It is by design that Jama Software passes all steps in this circumstance. For better or worse (in yours and Randy's case, worse), Jama Software is designed to pass the whole test if the final step is passed. The steps function assumes that steps are completed in numerical order. But Randy is right; at this point we don't have any plans to change this.

------------------------------
Kristina King
Jama Software
------------------------------
-------------------------------------------
Original Message:
Sent: 04-27-2017 01:11
From: Ken G
Subject: Jama Automatically Passes Skipped Test Steps

For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama Software will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama Software to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

------------------------------
Ken G
ASR
------------------------------

[Deleted User]

I can imagine that this functionality to auto pass is useful when reviewing requirements, but for testing, this is risky. I can't imagine a tester reviewed this functionality when it was implemented and said 'yes let's auto skip steps by default to save time.
It takes away from the detailed approach of testing and not missing items.

There are a few ways things function in Jama Software that doesn't make sense for testing. I understand people can approach testing differently, but in testing there are a few obvious things that are mandatory for testing (not skipping steps, providing more than just Passed or Failed for a test step - i.e descoped, or N/A)


------------------------------
Diana
------------------------------
-------------------------------------------
Original Message:
Sent: 05-05-2017 19:07
From: Kristina King
Subject: Jama Automatically Passes Skipped Test Steps

Hi Ken, 
It is by design that Jama passes all steps in this circumstance. For better or worse (in yours and Randy's case, worse), Jama is designed to pass the whole test if the final step is passed. The steps function assumes that steps are completed in numerical order. But Randy is right; at this point we don't have any plans to change this.

------------------------------
Kristina King
Jama Software
------------------------------
-------------------------------------------
Original Message:
Sent: 04-27-2017 01:11
From: Ken G
Subject: Jama Automatically Passes Skipped Test Steps

For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

------------------------------
Ken G
ASR
------------------------------

[Deleted User]

Hi Diana,

This feature was specifically designed to function this way, and like all features, there was real-world testing before it shipped. But this was many years ago, and the average Jama Software customer might have changed since then. I'm curious-is your company in the same position as Ken's, where Step order does not necessarily mean Execution order?

------------------------------
Kristina King
Jama Software
------------------------------
-------------------------------------------
Original Message:
Sent: 06-22-2017 21:29
From: Diana Matic
Subject: Jama Automatically Passes Skipped Test Steps

I can imagine that this functionality to auto pass is useful when reviewing requirements, but for testing, this is risky. I can't imagine a tester reviewed this functionality when it was implemented and said 'yes let's auto skip steps by default to save time.
It takes away from the detailed approach of testing and not missing items.

There are a few ways things function in Jama that doesn't make sense for testing. I understand people can approach testing differently, but in testing there are a few obvious things that are mandatory for testing (not skipping steps, providing more than just Passed or Failed for a test step - i.e descoped, or N/A)


------------------------------
Diana
------------------------------
-------------------------------------------
Original Message:
Sent: 05-05-2017 19:07
From: Kristina King
Subject: Jama Automatically Passes Skipped Test Steps

Hi Ken, 
It is by design that Jama passes all steps in this circumstance. For better or worse (in yours and Randy's case, worse), Jama is designed to pass the whole test if the final step is passed. The steps function assumes that steps are completed in numerical order. But Randy is right; at this point we don't have any plans to change this.

------------------------------
Kristina King
Jama Software
------------------------------
-------------------------------------------
Original Message:
Sent: 04-27-2017 01:11
From: Ken G
Subject: Jama Automatically Passes Skipped Test Steps

For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

------------------------------
Ken G
ASR
------------------------------

[Deleted User]

Hi Kristina, 
Step order does mean Execution order generally, but there is a scenario where for example I may need to check that multiple fields have a certain property. In this case, the test steps would have:
Test Step
1             Verify that x applies for the following fields
2              Field 1
3              Field 2
4              Field 3
5              ..etc

In this case I might verify Field 3 first because I am waiting for something for Field 2. I'd rather split out each field so they each get proper attention than bunching it all into one test case. This is a simple example, but you can see one case where this is applicable.
Something to consider in a future release.

Regards, 
Diana

[Deleted User]

I would suggest Jama Software to provide an option to either toggle the skip feature to auto pass the skipped steps or not. Default = not.

------------------------------
Thanks,
Ken G
------------------------------
-------------------------------------------
Original Message:
Sent: 06-25-2017 19:31
From: Diana Matic
Subject: Jama Automatically Passes Skipped Test Steps

Hi Kristina, 
Step order does mean Execution order generally, but there is a scenario where for example I may need to check that multiple fields have a certain property. In this case, the test steps would have:
Test Step
1             Verify that x applies for the following fields
2              Field 1
3              Field 2
4              Field 3
5              ..etc

In this case I might verify Field 3 first because I am waiting for something for Field 2. I'd rather split out each field so they each get proper attention than bunching it all into one test case. This is a simple example, but you can see one case where this is applicable.
Something to consider in a future release.

Regards, 
Diana

------------------------------
Diana
------------------------------
-------------------------------------------
Original Message:
Sent: 06-23-2017 11:38
From: Kristina King
Subject: Jama Automatically Passes Skipped Test Steps

Hi Diana,

This feature was specifically designed to function this way, and like all features, there was real-world testing before it shipped. But this was many years ago, and the average Jama customer might have changed since then. I'm curious-is your company in the same position as Ken's, where Step order does not necessarily mean Execution order?

------------------------------
Kristina King
Jama Software
------------------------------
-------------------------------------------
Original Message:
Sent: 06-22-2017 21:29
From: Diana Matic
Subject: Jama Automatically Passes Skipped Test Steps

I can imagine that this functionality to auto pass is useful when reviewing requirements, but for testing, this is risky. I can't imagine a tester reviewed this functionality when it was implemented and said 'yes let's auto skip steps by default to save time.
It takes away from the detailed approach of testing and not missing items.

There are a few ways things function in Jama that doesn't make sense for testing. I understand people can approach testing differently, but in testing there are a few obvious things that are mandatory for testing (not skipping steps, providing more than just Passed or Failed for a test step - i.e descoped, or N/A)


------------------------------
Diana
------------------------------
-------------------------------------------
Original Message:
Sent: 05-05-2017 19:07
From: Kristina King
Subject: Jama Automatically Passes Skipped Test Steps

Hi Ken, 
It is by design that Jama passes all steps in this circumstance. For better or worse (in yours and Randy's case, worse), Jama is designed to pass the whole test if the final step is passed. The steps function assumes that steps are completed in numerical order. But Randy is right; at this point we don't have any plans to change this.

------------------------------
Kristina King
Jama Software
------------------------------
-------------------------------------------
Original Message:
Sent: 04-27-2017 01:11
From: Ken G
Subject: Jama Automatically Passes Skipped Test Steps

For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

------------------------------
Ken G
ASR
------------------------------

[Deleted User]

Hi,
any update?
We find this behavior very problematic and I expect it to be the same for most users that are not in the Software industry.

Feature request> Test Run: option to disable Steps auto-fill
Steps in Test Runs were obviously develop for Software evaluation. Also by the name, it's clear that a sequence of test "steps" is assumed: when defining a step as PASSED or FAILED there is a feature that auto-fills previous steps as PASSED when undefined (NOT_RUN). This is great for clicks saving provided that steps are in sequence.
For Hardware test, I don't see the need to check steps of a sequence but to check the same Test Case under different external conditions (temperature, loads, supply voltages…). So a use Steps for my different condition but I'm then very annoyed by the auto-fill feature since the conditions were not necessarily defined in the order they are finally tested.
My proposal is then to add an option in the Test Case configuration to enabling/disabling the step auto fill. It would also be great to be able to customize the Step field and related table headers.

Ashley Ernst

Hello, thank you all for your ongoing feedback. In an effort to improve testing efficiency and outcomes, we've introduced new options in 9.5 which are relevant to this discussion:

Organization Administrators may configure how the system will behave when users skip test steps and don't assign them a status:

• Assign "Passed" to all skipped steps

• Do not assign status to any skipped steps

• Allow tester to select status for skipped step (i.e. ignore, pass, fail)

Instructions:

1. Navigate to Admin > Organization Admin > Item types
2. Select the Test Run item type in the left panel
3. Select the Status field in the right panel and click the pencil icon to edit
4. From the popup window, select the desired default behavior for completing steps out of order 
5. Click save

Organization Administrators may configure "Not applicable"/"N/A" as an additional status option. Once configured, testers will see the option to mark some or all steps in a run as 'Not applicable'. Any steps marked as 'Not applicable' will be excluded from the overall pass/fail result of the run.

Instructions:

1. Navigate to Admin > Organization Admin > Item types
2. Select the Test Run item type in the left panel
3. Select the Status field in the right panel and click the pencil icon to edit
4. From the popup window and under the status options, select the box for "Not Applicable (N/A)" status
5. Click save

We welcome any additional feedback on these new options, as we're are eager to understand the impact.

------------------------------
Ashley Ernst
CA
------------------------------
-------------------------------------------
Original Message:
Sent: 02-21-2020 01:39
From: Christian Binard
Subject: Jama Software Automatically Passes Skipped Test Steps

Hi,
any update?
We find this behavior very problematic and I expect it to be the same for most users that are not in the Software industry.

Feature request> Test Run: option to disable Steps auto-fill
Steps in Test Runs were obviously develop for Software evaluation. Also by the name, it's clear that a sequence of test "steps" is assumed: when defining a step as PASSED or FAILED there is a feature that auto-fills previous steps as PASSED when undefined (NOT_RUN). This is great for clicks saving provided that steps are in sequence.
For Hardware test, I don't see the need to check steps of a sequence but to check the same Test Case under different external conditions (temperature, loads, supply voltages…). So a use Steps for my different condition but I'm then very annoyed by the auto-fill feature since the conditions were not necessarily defined in the order they are finally tested.
My proposal is then to add an option in the Test Case configuration to enabling/disabling the step auto fill. It would also be great to be able to customize the Step field and related table headers.

------------------------------
Christian Binard
ON semiconductor
Belgium
------------------------------

Original Message:
Sent: 06-26-2017 02:27
From: Ken G
Subject: Jama Software Automatically Passes Skipped Test Steps

I would suggest Jama Software to provide an option to either toggle the skip feature to auto pass the skipped steps or not. Default = not.

------------------------------
Thanks,
Ken G

Original Message:
Sent: 06-25-2017 19:31
From: Diana Matic
Subject: Jama Automatically Passes Skipped Test Steps

Hi Kristina,
Step order does mean Execution order generally, but there is a scenario where for example I may need to check that multiple fields have a certain property. In this case, the test steps would have:
Test Step
1             Verify that x applies for the following fields
2              Field 1
3              Field 2
4              Field 3
5              ..etc

In this case I might verify Field 3 first because I am waiting for something for Field 2. I'd rather split out each field so they each get proper attention than bunching it all into one test case. This is a simple example, but you can see one case where this is applicable.
Something to consider in a future release.

Regards,
Diana

------------------------------
Diana

Original Message:
Sent: 06-23-2017 11:38
From: Kristina King
Subject: Jama Automatically Passes Skipped Test Steps

Hi Diana,

This feature was specifically designed to function this way, and like all features, there was real-world testing before it shipped. But this was many years ago, and the average Jama customer might have changed since then. I'm curious-is your company in the same position as Ken's, where Step order does not necessarily mean Execution order?

------------------------------
Kristina King
Jama Software

Original Message:
Sent: 06-22-2017 21:29
From: Diana Matic
Subject: Jama Automatically Passes Skipped Test Steps

I can imagine that this functionality to auto pass is useful when reviewing requirements, but for testing, this is risky. I can't imagine a tester reviewed this functionality when it was implemented and said 'yes let's auto skip steps by default to save time.
It takes away from the detailed approach of testing and not missing items.

There are a few ways things function in Jama that doesn't make sense for testing. I understand people can approach testing differently, but in testing there are a few obvious things that are mandatory for testing (not skipping steps, providing more than just Passed or Failed for a test step - i.e descoped, or N/A)


------------------------------
Diana

Original Message:
Sent: 05-05-2017 19:07
From: Kristina King
Subject: Jama Automatically Passes Skipped Test Steps

Hi Ken,
It is by design that Jama passes all steps in this circumstance. For better or worse (in yours and Randy's case, worse), Jama is designed to pass the whole test if the final step is passed. The steps function assumes that steps are completed in numerical order. But Randy is right; at this point we don't have any plans to change this.

------------------------------
Kristina King
Jama Software

Original Message:
Sent: 04-27-2017 01:11
From: Ken G
Subject: Jama Automatically Passes Skipped Test Steps

For example, when executing a test case with 8 test steps. Test steps 1, 2, 3 = Pass. Skip test steps 4 and 5 and pass step test 6. Jama will automatically mark test steps 4 and 5 as 'Pass'. I expect Jama to notify the tester that test steps 4 and 5 were skipped, and NOT allow test step 6 = pass.

------------------------------
Ken G
ASR
------------------------------