Test Runs - "Pass with Errors" concept.

[Deleted User]
[Deleted User] Member Posts: 2
edited March 13 in Peer-to-Peer Support Forum

Regarding Test Run, I don't understand what is 'Pass with Errors' useful to. And I am not comfortable with the idea of declaring that I have passed a test while it revealed errors. I suspect that we will want to prevent our users from checking this option.

Where could I find some use case examples for this option?
How could I make this option unusable/inaccessible to users?

Regards

Comments

  • Alessandro Valli
    Alessandro Valli Member, Data Exchange, Jama Connect Interchange™ (JCI) Posts: 787
    edited May 2020
    Hi Frederic,
    we mostly use 'Pass with Errors' as 'Fail accepted'. That are Fails, but with User override (accept it anyway ... so to say).
    Regards,
    Alessandro
    Systems Engineer
    SICK AG
  • [Deleted User]
    [Deleted User] Product Team Posts: 334
    edited May 2020
    Alessandro:

    You always contribute such great ideas and workflows! I granted our new "Collaborator" badge to your community profile!


    image
    Keep it up!

  • Alessandro Valli
    Alessandro Valli Member, Data Exchange, Jama Connect Interchange™ (JCI) Posts: 787
    edited May 2020
    Somehow related to this,

    what I am missing is a state (both Test Case Status, Test Run Status ) like "inconclusive" in addition to "blocked".
    • Blocked should be "not finished, but blocked"
    • Inconclusive should be "finished, but neither passed, nor failed"
    Best,
    Alessandro
    Systems Engineer
    SICK AG
  • Rory Duncan
    Rory Duncan Member Posts: 30
    edited June 2020
    Ideally, we should be able to provide our own pick list of criteria for test run statuses.

    RAD
  • [Deleted User]
    [Deleted User] Product Team Posts: 334
    edited June 2020
    Rory and Alessandro:

    The Test Center is an area of focus right now. Here is a list of Idea cards from other Community members in the new Ideation module so far: Test Center Idea Cards

    Take a look, vote on any ideas you like and make new ones!
  • Ashley Ernst
    Ashley Ernst Jama Staff, Product Manager Moderator Posts: 45
    edited November 2023

    We haven't released the ability to create custom statuses, however,  in 9.5 we released some options which may provide needed flexibility for you:

    Organization Administrators may configure how the system will behave when users skip test steps and don't assign them a status:

    • Assign "Passed" to all skipped steps

    • Do not assign status to any skipped steps

    • Allow tester to select status for skipped step (i.e. ignore, pass, fail)

    Instructions:

    1. Navigate to Admin > Organization Admin > Item types
    2. Select the Test Run item type in the left panel
    3. Select the Status field in the right panel and click the pencil icon to edit
    4. From the popup window, select the desired default behavior for completing steps out of order 
    5. Click save

    Organization Administrators may configure "Not applicable"/"N/A" as an additional status option. Once configured, testers will see the option to mark some or all steps in a run as 'Not applicable'. Any steps marked as 'Not applicable' will be excluded from the overall pass/fail result of the run.

    Instructions:

    1. Navigate to Admin > Organization Admin > Item types
    2. Select the Test Run item type in the left panel
    3. Select the Status field in the right panel and click the pencil icon to edit
    4. From the popup window and under the status options, select the box for "Not Applicable (N/A)" status
    5. Click save

    The product team would like to learn more about what other statuses would you add (if that capability were available) and how would you use them to improve efficiency/outcomes. Feel free to add any insights the product team will use to understand your use case here:

    https://community.jamasoftware.com/product-ideas-suggest/ideation-home/viewidea?IdeationKey=fd8701ec-d410-497b-bee7-858fc86923bf#ItemCommentPanel

    ------------------------------
    Ashley Ernst
    CA
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 06-17-2020 10:49
    From: Chloe Elliott
    Subject: Test Runs - "Pass with Errors" concept.

    Rory and Alessandro:

    The Test Center is an area of focus right now. Here is a list of Idea cards from other Community members in the new Ideation module so far: Test Center Idea Cards

    Take a look, vote on any ideas you like and make new ones!

    ------------------------------
    Chloe Elliott
    Jama Software
    Portland OR
    ------------------------------

    Original Message:
    Sent: 06-17-2020 05:53
    From: Rory Duncan
    Subject: Test Runs - "Pass with Errors" concept.

    Ideally, we should be able to provide our own pick list of criteria for test run statuses.

    RAD

    ------------------------------
    Rory Duncan
    NASA
    Kennedy Space Center FL
    3218612345

    Original Message:
    Sent: 05-26-2020 08:30
    From: Alessandro Valli
    Subject: Test Runs - "Pass with Errors" concept.

    Somehow related to this,

    what I am missing is a state (both Test Case Status, Test Run Status ) like "inconclusive" in addition to "blocked".
    • Blocked should be "not finished, but blocked"
    • Inconclusive should be "finished, but neither passed, nor failed"
    Best,

    ------------------------------
    Alessandro
    Systems Engineer
    SICK AG

    Original Message:
    Sent: 05-26-2020 07:57
    From: Chloe Elliott
    Subject: Test Runs - "Pass with Errors" concept.

    Alessandro:

    You always contribute such great ideas and workflows! I granted our new "Collaborator" badge to your community profile!

    image
    Keep it up!



    ------------------------------
    Chloe Elliott
    Jama Software
    Portland OR

    Original Message:
    Sent: 05-25-2020 06:59
    From: Alessandro Valli
    Subject: Test Runs - "Pass with Errors" concept.

    Hi Frederic,
    we mostly use 'Pass with Errors' as 'Fail accepted'. That are Fails, but with User override (accept it anyway ... so to say).
    Regards,

    ------------------------------
    Alessandro
    Systems Engineer
    SICK AG

    Original Message:
    Sent: 05-25-2020 06:18
    From: Frederic GARFAGNI
    Subject: Test Runs - "Pass with Errors" concept.

    Regarding Test Run, I don't understand what is 'Pass with Errors' useful to. And I am not comfortable with the idea of declaring that I have passed a test while it revealed errors. I suspect that we will want to prevent our users from checking this option.

    Where could I find some use case examples for this option?
    How could I make this option unusable/inaccessible to users?

    Regards


    ------------------------------
    Frederic GARFAGNI
    Fresenius Kabi
    Development Methods & Tools
    GRENOBLE - France
    ------------------------------
    Ashley Ernst
    [Pronouns]
    Product Manager
    Jama Software

Categories