Blogs

How Does the Status of a Test Run Factor Into the Status of a Test Case?

By Carmen Santos posted 11-20-2015 07:05

  


Test Plans have no status. Test Case Status is a field derived from the lowest status of the last executed Test Run in any active Test Plan (regardless of the order of the Test Run's creation). This means that archived Test Plans will not be considered for status evaluation.

https://higherlogicdownload.s3.amazonaws.com/JAMASOFTWARE/BlogFiles/41fc955c-43c1-4f8f-a146-e989521b42b2/RackMultipart20151120-25353-z4doi7-Screen_Shot_2015-11-20_at_9.35.50_AM_inline.pngTo keep irrelevant plans from affecting the Test Case status, you can archive superfluous Test Plans. To do this, right-click Test Plan > Archive. You can unarchive the Test Plans in the same way as they still show up in the Test Plans list, grouped as Archived Test Plans.

https://higherlogicdownload.s3.amazonaws.com/JAMASOFTWARE/BlogFiles/41fc955c-43c1-4f8f-a146-e989521b42b2/RackMultipart20151120-1793-a56h5a-Screen_Shot_2015-11-20_at_9.38.21_AM_inline.pngIf no Test Runs are associated with the Test Case, its status is NOT SCHEDULED.

Calculation Conditions

The Test Case status calculation is based on the most recent Test Run from all Test Plans where the Test Case is present (except archived Test Plans).

This means a Test Case can fail even when the most recent Test Run from all Test Plans is passed.

Include Unexecuted Test Runs in Status Calculations 
Under the Administration Organization Detail panel, there is an option to "Include Unexecuted Test Runs in Status Calculations".

https://higherlogicdownload.s3.amazonaws.com/JAMASOFTWARE/BlogFiles/41fc955c-43c1-4f8f-a146-e989521b42b2/RackMultipart20151120-26060-1t2uyii-Screen_Shot_2015-11-20_at_9.57.08_AM_inline.pngBy default, this calculation setting is checked for existing and new organizations, and unexecuted test runs are included in the calculation. To include only executed test runs in the calculation this flag needs to be unchecked. However, this is not retroactive.

*It is important to note that if you generate a Test Plan report, the Test Case Status in that report will be based on the most recent run in any plan, not that plan specifically. You would need to archive a test plan if you do not want the status of its runs to affect the Test Case Status.

More on Test Case Status

Test case status is an automatically calculated status that indicates the state of the last edited test run associated with that test case.

By default, this status is calculated using all available test runs associated with the test case, including runs that aren't scheduled. An organization administrator can disable this option so that unexecuted runs aren't included in the calculation.

The status is calculated in one of these methods:

  • If the test case is used in only one test plan, the latest status is reported on the test case
  • If the test case is used in multiple test plans, test case status shows the most urgent test run status based on the following hierarchy, regardless of when it ran: Not Scheduled, Failed, Blocked, Scheduled, Passed.

The status is visible in Single Item View of each test case.

Note

Because test case status is automatically calculated, it can't be manually updated when editing a test case.


Tip

If a test case is used in multiple plans and it's picking up the status of a plan that's no longer relevant, one option is to archive that plan. Doing so removes that plan's test run statuses from the test case status calculation.

You can also use archiving to trigger a recalculation of all test cases in a plan. When a user archives and then un-archives the test plan, this results in a re-calculation of the status for all test cases in that plan. This can be useful if you change the admin setting to include unexecuted test runs (which isn't retroactive) and want to recalculate test case status with the new setting.

8 comments
694 views

Comments

01-05-2016 16:41

If there is something that doesn't seem accurate, I'd like to get a bug filed. So as we are talking through this, you're helping me pin down what is off. I'm basing my statements off the chart in the user guide about what unchecking the setting does.



According to it, in scenario 1, the Test Case Status will be reflected as "Scheduled" in the event that there are only Not Run/Unexecuted runs. 
So that's where I thought SOS-BUG-356 could be coming into play—even though the "Not Run" cycle was deleted, the Test Case Status is reflected as "Scheduled". Or at least, this would be the case given Scenario 3, flag checked.
But since the flag is unchecked*, it should really be showing as Failed, in Scenario 3 Flag unchecked. So I think you're right, that there is something wrong with this flag in the admin setting. I suppose that's what I need to try to reproduce and file a bug about. In the end, I guess it's best we forget I ever brought up that flag ;)

*Note that this weirdness might be explained given that the flag isn't retroactive, however, I did archive & unarchive the pertinent plans according to the suggestion in the user guide. 

01-05-2016 11:50

I agree, this is complex.  We wrestled with it for some time before the workaround was discovered (which is why I decided to share it).  Your initial state seems to suggest there is a flaw in the admin setting also.  Having no other runs with status other than scheduled, there should be no gray in your example chart at all because scheduled is  "Unexecuted".  Correction of SOS-BUG-356 is not likely to fix this as the chart is incorrect before that bug comes into play.

01-04-2016 19:16

In my example above, I have the "Include Unexecuted Test Runs in Status Calculations [Not Retroactive]:" unchecked. The items showing as "Scheduled" do not have runs with statuses besides "Scheduled."

I tried what you suggested and created a new cycle from my 5 Failed runs, which I then deleted. I refreshed the chart and the number of items doesn't change in the pie chart—however, the wrong thing happened. The items should have reverted back to Failed after deleting the newest cycle, however, they stayed Scheduled. (And with the setting checked, Scheduled should not have come to play for items that had a different test status).

So it appears that in the end this particular setting doesn't matter. I think this is a ramification of SOS-BUG-356, where deleting/archiving a test plan or deleting a cycle does not update the calculation. Unchecking that setting should prevent "Scheduled" from masking the previous failed/blocked/passed results.

In the end, Benjamin, I think your workaround is best for what you're attempting to accomplish, and I know better than to ask about this particular setting. (Or at least until SOS-BUG-356 is fixed.)

01-04-2016 17:55

We need to see % progress against the entire test task, and each test should only be represented once.  The problem is the test case status does not update (revert) when a newer run or cycle is deleted.
In the example above, create a new cycle using just the items with a failed previous result.
Delete that new cycle.
Now look at your pie chart.
Those failed items now appear scheduled even though they aren't, and their most recent result of failed is obscured.
Won't unchecking the "show only executed" make the chart not show those items at all because the most recent result was not executed?

01-04-2016 15:05

I think you bring up probably the most complicated part of this setting. If you uncheck the setting, your Test Case Status will be "Scheduled" for those particular cases that have runs only in a Scheduled status, and they will appear in the pie chart.


If any of the cases have statuses from a previous plan that are not scheduled, those cases will show up as the previous status. So in a way you're right. Unchecking the setting would make cases that had Failed previously but have a new run scheduled still show up as Failed. So while the items don't disappear from the widget, their most recent run status of Scheduled wouldn't be shown if there were another previous (actually run) status. So while there is information missing, it's not entire cases.

I hope I didn't complicate this further.

01-04-2016 10:22

We use a pie chart widget on the home screen of every project, based on a filter for test cases that have a run in a specific (release) plan.  The chart provides a quick status overview without having to run a report.  Wouldn't unchecking that setting just remove the items from the pie charts altogether because their most recent status is "scheduled" (unexecuted)?

12-28-2015 19:33

Thanks for pointing out that workaround, Benjamin. I'd be curious if others have run into this issue, or if it's an edge case. Is there a reason you don't uncheck the "Include Unexecuted Test Runs in Status Calculations" setting?

12-23-2015 11:07

We archive all but the current plan.  When a test run is added to a new cycle, and subsequently deleted (or the cycle itself is deleted), the status for the test case retains the "scheduled" status (meaning not tested).  It should revert to what it was (the most recent result).  We found a "workaround" - reopen that run in the most recent cycle and save and close.  This basically recommits the same result (and returns the status to what it should be), but of course you get a new time stamp associated with that result.