Integrate your test automation framework with Azure Test Plan
Ever wondered how to make your automated tests written in Java/JavaScript/Python, integrate with Azure Test Plan 🤔? If yes, then look no further!! In this article we will look into a step by step process of how we can tweak our existing test automation framework to interact with Azure Test Plan. Microsoft has always loved C# and Visual Studio, so in Azure DevOps, the integration of C# tests with Test Plan come out of the box. For everything else, we need to find our own ways..
In order to achieve this we will make use of the super cool Azure DevOps APIs which by the way are very nicely documented by Microsoft but are way too many and way too confusing 🤯. Finding the exact API we need, is like finding a needle in a haystack.
The objective :
- Create a new “Run” in Azure DevOps when the automated tests run as part of Build or Release pipeline
- Pull the manual test cases, which correspond to the automated test cases, into this new run.
- Mark the outcome of the “executed” test cases in Test Plan as Passed or Failed, or leave them as Active (default) if they are not included for execution.
- If a testcase fails, automatically create a bug and associate it with the failing test case. This can be viewed in the “Run” result.
- If the testcase passes on a rerun or in the next iteration, close the defect and automatically mark the test case as Passed.
The API Solution :
- Get Test Plan ID
Endpoint :
https://dev.azure.com/yourOrganizationName/yourProjectName/_apis/test/plans?api-version=5.0Method :
GETJsonPath to get Test Plan ID :
$.value.[?(@.name == 'yourPlanName')].id
2. Get Test Suite ID
Endpoint :
https://dev.azure.com/yourOrganizationName/yourProjectName/_apis/test/plans/planID/suites?api-version=5.0Method :
GETJsonPath to get Test Suite ID :
$.value.[?(@.name == 'yourSuiteName')].id
planID-is available from step1
3. Get Test Case ID
Endpoint :
https://dev.azure.com/yourOrganizationName/yourProjectName/_apis/test/plans/planID/suites/suiteID/points?api-version=5.0Method :
GETJsonPath to get Test Case ID :
$..[?(@.name == 'yourTestCaseName')].id
planID-is available from step1
suiteID-is available from step2
4. Get Test Point ID
Endpoint :
https://dev.azure.com/yourOrganizationName/yourProjectName/_apis/test/plans/planID/suites/suiteID/points?testCaseId=tcID&api-version=5.0Method :
GETJsonPath to get Test Point ID :
$.value.[0].id
planID-is available from step1
suiteID-is available from step2
tcID-is available from step3
5. Create Test Run
Endpoint :
https://dev.azure.com/yourOrganizationName/yourProjectName/_apis/test/runs?api-version=5.0Method :
POSTContent-Type :
application/jsonSample Payload:
{"name":"runName","plan":{"id":planID},"pointIds":[pointID]}JsonPath to get Test Run ID :
$.id
planID-is available from step1
pointID-is available from step4
6. Get Test Result ID
Endpoint :
https://dev.azure.com/yourOrganizationName/yourProjectName/_apis/test/runs/runID/results?api-version=6.0-preview.6Method :
GETJsonPath to get Test Result ID :
$.value.[0].id
runID-is available from step5
7. Create Bug (Optional step)
Endpoint :
https://dev.azure.com/yourOrganizationName/yourProjectName/_apis/wit/workitems/$bug?api-version=5.0Method :
POSTContent-Type :
application/json-patch+jsonSample Payload:
[{"op": "add","path": "/fields/System.Title","from":null, "value":"titleOfBug"}]JsonPath to get Bug ID :
$.id
8. Update Results in Test Run
Endpoint :
https://dev.azure.com/yourOrganizationName/yourProjectName/_apis/test/runs/runID/results?api-version=6.0-preview.6Method :
PATCHContent-Type :
application/jsonSample Payload if Passed:
[{ "id": resultID , "outcome": "PASSED" ,"state": "Completed", "comment": "Execution Successful" }]Sample Payload if Failed:
[{ "id": resultID , "outcome": "FAILED" ,"state": "Completed", "comment": "Execution Failed", "associatedBugs": [{"id":bugID}]}]
runID-is available from step5
resultID-is available from step6
bugID-is available from step7
And voila!! There you go.. These are all you need to be able to report your automated test case status to Azure Test Plan.
When all test cases pass, your Test Plan looks like this :
And if any test case fails it looks like this :
I was able to successfully integrate this set up with Java-Cucumber tests and Robot-Python tests. Likewise this can be extended to any JS framework like WebDriverIO or Cypress.
One thing to note however is , the above setup shows the examples with 1 Test Plan, 1 Test Suite and 1 test case to illustrate how the process works. However based on the needs, this can be modified to handle multiple Plans, Suites and Test Cases.
To view a simple python implementation of this you can check my GitHub link:
To view the complete Microsoft Documentation on Azure DevOps check this link:
