Support

Expand all | Collapse all

Updating Test Run via REST API challenges...

  • 1.  Updating Test Run via REST API challenges...

    Posted 02-03-2017 14:12
    Hi,

    We are currently using on-premise Jama 8.4.1; the discussion below applies to my knowledge of this specific version of Jama.

    It's clear that the Test Run type is a bit of a unique animal in Jama.  This makes it a bit of a challenge to work with via REST API.  For all other item types, we can programmatically query the item type (via GET /itemtypes/{id}) to determine each field's type (and the associated pick list, as appropriate) as well as other attributes such as required, read only, etc.  However, for the Test Run, the ONLY PUT call (PUT /testruns/{id}) is used to "Update the execution results... " which essentially has its own set of "unknowable" settings for the field attributes.  For example, name and description fields are Read Only (though the REST call does not return an error when these values are changed).  On the other hand, duration and testRunStatus are not Read Only.  And assignedTo is Required.  None of these match with the field attributes of the Test Run item type retrieved through the GET /itemtypes/{id} call.

    I do understand that the intent is to match the functionality of the Test Run as it is executed by a user in the Jama UI.  Matching the functionality is not really an issue; rather, the issue is that there is no way to know programmatically which fields are RW/RO, required when executing the test run.  Of course we can hard-code the information but that adds overhead as it would be a custom code path just for Test Run and if things change it's something we have to manually update.

    Additionally, there is value in being able to update Test Run fields such as the name and description.  For example, the test configuration can be pushed into the Test Run description at the time the test is executed via automation.

    One possible solution to this would be to use the PUT /testruns/{id} call to allow updating the Test Run in a way that aligns with the ability to "Edit" the Test Run in the UI (and would also provide consistent behavior across all similar REST calls) and then use a new call PUT /testruns/{id}/execute specifically for the execution behavior.  It would still be necessary to provide some method to retrieve field attributes which are unique to the test execution behavior.  Maybe it could be built into the current item types structure, with "fields" being default ("Edit") behavior and "fields1" through "fieldsx" being alternate behavior?

    Of course, it may be too late to change the behavior of the PUT /testruns/{id} call, so another alternative is to allow normal "Edit" behavior through the PUT /items/{id} call for all item types, rather than excluding Test Management types from that call.

    If you are already looking at solutions to this kind of problem, I would be interested to hear about it.  If not, I would be happy to submit an enhancement request instead.

    Thanks,
    Mike

    ------------------------------
    Success always occurs in private, and failure in full view. ~Steven Wright
    ------------------------------


  • 2.  RE: Updating Test Run via REST API challenges...

    Posted 02-03-2017 14:33
    Edited by Sam Detweiler 02-03-2017 14:34
    You can use the itemtypes api to get the field details for the testrun type... then u have all the info you need.

    also note, that if the testcase contains steps, then to update the testrun, you MUST supply a result status for each STEP.

    ------------------------------
    Sam
    ------------------------------



  • 3.  RE: Updating Test Run via REST API challenges...

    Posted 02-07-2017 12:28
    Hi Sam,

    Unfortunately, what you say is not correct and is actually the point of my post.

    If I get the Test Run item type details through GET /itemtypes/{id}, the returned data indicates the Name field is "not read only" (aka it is read/write):

    { "id": 1287,
    "
    name": "name",
    "
    label": "Name",
    "
    fieldType": "STRING",
    "
    readOnly": false,
    "
    required": true,
    "
    triggerSuspect": false,
    "
    synchronize": false
    }


    Yet there is not one API call that allows me to update the Name field of the Test Run.

    Similarly, the data returned indicates that the duration is read only, yet with the PUT /testruns/{id} I am able to update the duration without issue:

    { "id": 1297,
    "
    name": "duration",
    "
    label": "Duration",
    "
    fieldType": "TIME",
    "
    readOnly": true,
    "
    required": false,
    "
    triggerSuspect": false,
    "
    synchronize": false
    }


    So my point is that there is no API call that honors the item type details returned in the Test Run itemtypes call, and there is no way to get the Test Run details that specify what parameters are needed (or are allowable) in the PUT /testruns/{id} call.

    If you have examples of how to update the name and description of the Test Run (as the itemtype details indicate I can) or examples of how to determine which fields are r/w in the PUT /testruns/{id} call, then I would very much like to see them so we can correct our understanding.

    Thanks,
    Mike

    ------------------------------
    99 little bugs in the code. 99 little bugs in the code. Take one down, patch it around. 127 little bugs in the code... ~Anonymous
    ------------------------------



  • 4.  RE: Updating Test Run via REST API challenges...

    Posted 02-07-2017 13:05
    yep, looks like the itemtype details are incorrect.. I'd call that a bug. (in the yaml spec, not the api itself)

    I wouldn't expect to update the name and description fields, as they are copied from the testcase the testrun is cloned from.



    ------------------------------
    Sam
    ------------------------------



  • 5.  RE: Updating Test Run via REST API challenges...

    Posted 02-13-2017 13:43
    Hi Sam,

    The ItemType details do match what is possible through the UI if the user decides to edit the Test Run (rather than Executing it).  So it's not really a "bug" as such; rather, there are two different behaviors a user can address in the UI but only one of them is supported in the API.  The desired outcome of this discussion is that the API also supports both behaviors.

    In our situation, the test case is not always absolutely specific about what components to use when testing.  For example, memory, hard drive, display, software drivers may not be specified in the test case description.  However, when the actual test is run, we would like to record the exact components used.  This information is not really a "Test Step Result" nor is it an "Actual Result"; rather, it is part of the Test Run Description.  With our automation, we would like to be able to update the Test Run Description so that the exact Test configuration is captured independent of the Test results but still part of the Test Run.

    Mike

    ------------------------------
    Nobody will ever win the Battle of the Sexes. There's just too much fraternizing with the enemy. ~Henry Alfred Kissinger
    ------------------------------



  • 6.  RE: Updating Test Run via REST API challenges...

    Posted 02-23-2017 15:26
    I thought I had replied to this discussion, but I don't see my comment here.

    This is an area we are currently looking at and will be introducing some small updates to handle some of the gaps discussed here. One problem we started to work on, and backed off for now, was the testrun item type configuration lining up with the PUT and PATCH. The original goal of the testing resources was to align with the front which in turn means a big of black magic. We are reviewing our options now such that the API would be able to update all editable (or read only set to editable via API) fields on a testrun and not attempt to match the UI execute path. This is problematic as it might turn into a breaking change, but would definitely change behavior that could result in data being improperly updated.

    So look forward to some small changes, such as resetting a testrun from recent test case version while we continue to review our options including versioning the API.

    ------------------------------
    Steve Gotsch Product Manager
    Jama Software
    ------------------------------



  • 7.  RE: Updating Test Run via REST API challenges...

    Posted 06-20-2017 12:21
    I've had difficulty updating the testRunStatus of a test run. I can update the testRunStatus field using the PATCH /testruns/{id} endpoint in the Swagger tool, but I am unable to do the same thing using the Python requests module. I noticed that the Swagger tool includes a jama-csrf-token, but I was unable to figure out how to get or generate this token. I tried generating and using an OAuth2 access_token, but that didn't help either. At the end of the day, I get a 200 OK status back, but the PATCH doesn't change anything.

    ------------------------------
    Connor Johnson
    ------------------------------



  • 8.  RE: Updating Test Run via REST API challenges...

    Posted 06-20-2017 12:36

    When I updated testruns with results, I used PUT

     

    Sam

     






  • 9.  RE: Updating Test Run via REST API challenges...

    Posted 06-20-2017 13:22
    Hi Sam,

    I think you are talking about the actualResults field of the test run item type, which according to the GET /itemtypes/{id} API read/writable. I'm talking about the testRunStatus field, which is not read/writable, and does not allow API overwrite. Here are the details for the actualResults field, and the testRunStatus field:

    {
    "fieldType": "TEXT",
    "id": 289,
    "label": "Actual Results",
    "name": "actualResults",
    "readOnly": false,
    "required": false,
    "synchronize": false,
    "textType": "RICHTEXT",
    "triggerSuspect": false
    },...
    {
    "fieldType": "TEST_RUN_STATUS",
    "id": 293,
    "label": "Status",
    "name": "testRunStatus",
    "readOnly": true,
    "readOnlyAllowApiOverwrite": false,
    "required": false,
    "synchronize": false,
    "triggerSuspect": false
    },...

    Nevertheless, I can use Swagger to update the testRunStatus, but I can't seem to reproduce that result using curl or Python.


    ------------------------------
    Connor Johnson
    ------------------------------



  • 10.  RE: Updating Test Run via REST API challenges...

    Posted 06-20-2017 13:29

    There is a json structure of fields, which contain the duration, assignedTo, actualResults AND for single step testcycles, the testRunStatus field, OR for multi-step testcycles, the testRunsSteps field.

     

    Sam