Azure DevOps Self hosted Agents on Kubernetes

Scale up or down as per your demand!!

Hello there! After a break of around 6 months, I have decided to write again. This time, its about a super cool Azure DevOps hyperscale feature.

Have you been in situations where your corporate rules don’t allow you to run build/test pipelines on Azure provided agents? Or a situation where you want to execute multiple pipelines in parallel? If you are tired of provisioning Virtual Machines by now, then look no further…

In this article we will discuss how we can scale up and down our pipeline execution leveraging the power of Azure Kubernetes Cluster.

Selfhosted Agents — Hyperscaler

Prerequisites for this pipeline to run successfully :

1. Associate the correct Azure Subscription to your Azure DevOps Organization.
2. Create an Azure Container Registry (ACR) with the same subscription.
3. Create an Azure Kubernetes Cluster with the same subscription.
4. Create an Agent Pool in your Azure DevOps project following this.

My Resource Group contains these :

Let’s create some additional prerequisites before getting down to business.

In order to connect to the Azure Container Registry from Azure DevOps we can either create a Service connection or use docker login command. We will use the second option here. From Azure CLI enter the following command :

az acr scope-map create -n azureagentregistryscope -r SelfHostedAgentRegistry — repository selfhosted-agents content/write content/read

This will create a scope-map by the name of azureagentregistryscope with read/write access to the container registry which we created. In this case the name of the Registry is SelfHostedAgentRegistry and the repository inside the registry is selfhosted-agents.

After this enter the following from the CLI :

az acr token create --name acrToken --registry SelfHostedAgentRegistry --scope-map azureagentregistryscope --expiration 2025-12-31T12:59:59Z

This will generate a token which will help us login to the ACR. After running this command you should see a message like this :

Please store your generated credentials safely. Meanwhile you can use it through docker login selfhostedagentregistry.azurecr.io -u acrToken -p <tokenvalue>

We will use this tokenvalue later on in our build pipeline.

Now navigate to Azure DevOps and to your Project, and create a service connection for Azure Kubernetes Service:

Project Settings → Service Connections → Create New Service Connection → Choose Kubernetes → Choose Azure Subscription → Select Subscription , Cluster Name and Namespace → Give it a name and click [Save]

Now that we have finished creating the pre requisites, let’s move on to creating the pipeline in Azure DevOps.

I have provided all the files you need to create the pipeline in the following GitHub Repository :

ghoshasish99/AzureDevOps-SelfHostedAgents-AKS

Let’s quickly go through the files.

1. Dockerfile — This uses a base image of the microsoft/vsts-agent .If you need additional softwares you can add the commands to download and install here. By default standard softwares like nodeJS, java, python, etc. are available. To view this Dockerfile click here.
2. start.sh — This file is called inside the Dockerfile and is responsible for starting up the agents. It requires the name of the Agent Pool which we created above in step 4. It also needs the URL of your Azure DevOps Organization : https://dev.azure.com/<name of your organization>. Finally it requires your Personal Access Token (PAT) with which it can attach the agents to your organization. You can create a PAT following this. To view this start.sh file click here.
3. ReplicationController.yml — This is the Kubernetes manifest file which is responsible to deploy the agents in the Pods. The number of agents to be spun up is determined by the replicas : n tag. We also need to specify the imagePullSecrets in this file which is needed by Kubernetes to authenticate the pull from the ACR. Creation of this secret is a one time activity and can be achieved using this command from Azure CLI.

kubectl create secret docker-registry acr-secret --docker-server=<Registry URL> --docker-username=<your acr username> --docker-password=<your acr password>

acr-secret is the name of the secret that I created and is referred to in the file. To view this file click here.

4. azure-pipelines.yml — This file is to start the azure pipeline which will perform all the activities above. View this file here.

This build pipeline is divided into 2 stages :

The first stage is to build the docker image of the agents and push it to ACR. The second stage is to pull the images and deploy them in AKS thus spinning up the Selfhosted agents

In the first stage, there is only one task :

Build the docker image and push to Azure Container Registry.

- task: CmdLine@2
  displayName: 'Build and Push Image'
  inputs:
    script: |
     docker build -t agents:latest .
     docker login $(acrRegistry) --username $(acrusername) -- password $(azureACRToken)
     docker tag agents $(acrRegistry)/$(acrRepository)
     docker push $(acrRegistry)/$(acrRepository)

In the second stage, there are primarily 3 tasks :

1. Login to Azure Kubernetes Service using the Service Connection

- task: Kubernetes@1
  inputs:
    connectionType: 'Kubernetes Service Connection'
    kubernetesServiceEndpoint: '$(AKSserviceConnectionName)'
    command: 'login'

2. Replace some variables which were used in the ReplicationController.yml like AZP_POOL_NAME, AZP_URL_VALUE, AZP_TOKEN_SECRET etc. with Pipeline Variables. This obviously means that we need to create some pipeline variables like this.

3. The third task is to deploy the ReplicationController.yml

task: CmdLine@2
displayName: 'Deploy'
inputs:
   script: kubectl apply -f ReplicationController.yml

And Voila !!!

After the pipeline execution is complete, you should see new Agents available in the Agent Pool that you had created earlier.

Self hosted agents running on Kubernetes Cluster

Now to use these agents in any subsequent pipeline, you can simply use the below syntax in your YAML

trigger:
- main
pool: MySelfhostedAgentPool
steps:
- script: |
    python -m pip install --upgrade pip
    pip install pytest
    if [ -f requirements.txt ]; then pip install -r requirements.txt; 
    fi
  displayName: 'Install dependencies'

You can find the working Azure pipeline here :

Azure Pipeline

This article is inspired by this tutorial :
https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops