How to integrate CITS with Azure DevOps Build Pipeline

This document talks about how you can seamlessly integrate CITS into Azure DevOps and run your automated test cases as part of CI — CD.

CITS (Cognizant Intelligent Test Scripter) is an open source scriptless test automation tool for Web, Mobile and API automation. You can find the Github link here

We would walk through the 2 main techniques by which we can run CITS test cases in Azure DevOps.

1. Azure Build Pipeline.
2. Azure Releases.

In this article we will focus on Azure Build Pipeline. For Azure Release refer to this article

Before we get started, few things to know about Microsoft Azure DevOps :

Now let’s get started…

Chapter 1 : Azure Build Pipeline

1. Version control your CITS scripts.

You can use any of the following options. Usually the client will have a fixed directive on which Version Control tool to use within the organization.

For this demo/user guide, I have used GitHub. Check out this repository here .

2. Configure an Azure Build Pipeline

a. Log in to your Azure — Organization.
b. Select Pipelines >> Pipelines

3. Connect to the version control system

The [YAML] icon next to the version control options indicate that ADO expects a YAML (YAML Ain’t Markup Language) file inside your repository which determines how the pipeline should look like. This concept is generally referred to as “Pipelines as Code”

Below is an example of how a typical YAML for CITS execution would look like :

pool:
 vmImage: 'ubuntu-latest'
trigger:
- master
steps:
- task: CmdLine@2
  inputs:
     script: 'chmod -R 755 ./'
  displayName: 'Set Permissions'
- task: CmdLine@2
  inputs:
      script: './Run.command -run -project_location $(ProjectLocation) -release $(ReleaseName) -testset $(TestSetName)'
      workingDirectory: '$(System.DefaultWorkingDirectory)'
  displayName: 'Execute CITS Test'
- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'NUnit'
    testResultsFiles: '$(ProjectLocation)/Results/TestExecution/$(ReleaseName)/$(TestSetName)/Latest/azure.xml'
    failTaskOnFailedTests: true
    testRunTitle: 'API Test Run'
  displayName: 'Publish Reports'

You can find this file in the GitHub repository (https://github.com/ghoshasish99/CITS) mentioned above.

In a separate document I will discuss about how to create this YAML from the scratch in ADO. For now let’s proceed, with the assumption that this YAML is already present in your Repository.

4. As soon as you connect to the Version Control system, ADO recognizes the YAML and automatically configures the pipeline for you. You are then taken to the Review segment of the Pipeline where you can update the YAML file directly or add ADO tasks by clicking on [Show assistant] (as shown below)

5. Use of Build Variables

Before we talk about the Variables, lets take a deep dive into the YAML components and understand each section in details :

i. This section tells us what kind of ADO agent we want to run this Build. In this example, I have selected ‘ubuntu’. But you can go for Windows or MacOS options too.

pool:
 vmImage: 'ubuntu-latest'

ii. This section tells us which Branch of your repository will be responsible for triggering the build

trigger:
- master

iii. This Command Line Task sets permissions to the workspace where the contents of the Version Control system (specified branch) are checked in to the ADO agent.

steps:
- task: CmdLine@2
  inputs:
     script: 'chmod -R 755 ./'
  displayName: 'Set Permissions'

iv. This Command Line Task is responsible for the execution of CITS scripts
If you notice carefully, there are 4 variables which are part of this task.

3 user defined variables :

$(ProjectLocation)
$(ReleaseName)
$(TestSetName)

1 Predefined System variable :

$(System.DefaultWorkingDirectory)

We will talk about these variables in a short while.

v. This PublishTestResults Task is responsible for publishing the CITS Execution results in ADO. By default CITS generates a json report in addition to HTML reports. So we need to ensure that CITS generates Nunit or mocha style reports which can be read and parsed by ADO and published in the Tests section which we will cover in a later section.

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'NUnit'
    testResultsFiles: '$(ProjectLocation)/Results/TestExecution/$(ReleaseName)/$(TestSetName)/Latest/azure.xml'
    failTaskOnFailedTests: true
    testRunTitle: 'API Test Run'
  displayName: 'Publish Reports

The variables which I have defined for this build are as follows :

Now you are all set to run your Build by clicking on the Run button.

If everything goes well, you should see all the tasks have completed 😊 as shown below.

If you navigate to the Tests section you should see the complete Test Result details :

Some Handy Links :

1. Azure DevOps Project : https://dev.azure.com/401532/CITS/
2. GitHub Repository : https://github.com/ghoshasish99/CITS

References :

1. Azure DevOps Documentation :
https://docs.microsoft.com/en-us/azure/devops/?view=azure-devops

2. CITS GitHub Repository :
https://github.com/CognizantQAHub/Cognizant-Intelligent-Test-Scripter

3. YAML syntax documentation :
https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html