Running Azure DevOps Self-hosted agent on AKS without using PAT

PAT(Personal Access Token) has long been the prevalent authentication method for DevOps runners or agents, and the major purpose is to replace the usage of user credentials with a token-based approach to minimize the risk of credential spill. PAT is certainly useful in this aspect, but it is essentially nothing more than a series of random characters that inherits the same security concerns as password. On Azure, it’s important to note that Managed Identity is the preferred method for authentication/authorization. I have covered this topic in several articles before— Safely access Azure Kubernetes Service in GitHub Action with AAD Federated Identity and CI/CD Java apps securely to Azure Kubernetes Service with GitHub Action — Part 1

Thanks to an announcement back in March 2023— “Introducing Service Principal and Managed Identity support on Azure DevOps”, it became possible to utilize Managed Identity not only for GitHub Actions but also Azure DevOps Pipelines, eliminating the need for PAT. In this article, I will demonstrate how to set up AKS workload identity with an Azure DevOps Self-hosted agent, effectively eliminating the use of PAT in Azure DevOps pipelines

Prepare AKS Workload Identity

I will skip the AKS configuration part as it’s covered in my previous article — Using Managed Identity with Azure AD Workload Identity on AKS for Java developers. In short, AKS should be enabled with two add-ons, oidc issuer and workload identity.

We will start with creating managed identity.

$ az identity create - name "ado-agent-mi" - resource-group "sandbox-rg"
{
 "clientId": "07e18ba0-***,
 "id": "/subscriptions/***/resourcegroups/sandbox-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ado-agent-mi",
 "location": "eastus",
 "name": "ado-agent-mi",
 "principalId": "e4265182-***",
 "resourceGroup": "sandbox-rg",
 "systemData": null,
 "tags": {},
 "tenantId": "a6f0e660–***",
 "type": "Microsoft.ManagedIdentity/userAssignedIdentities"
}

Create an service account which is tied to managed identity.

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
 annotations:
   azure.workload.identity/client-id: 07e18ba0-***
 name: ado-agent-sa
 namespace: default
EOF

Federated credential with managed identity and oidc issuer.

$ az identity federated-credential create - name ado-agent-mi-fi - identity-name ado-agent-mi - resource-group sandbox-rg - issuer https://eastus.oic.prod-aks.azure.com/a6f0e660-***/f94fee6b-***/ - subject system:serviceaccount:default:ado-agent-sa - audience api://AzureADTokenExchange

Workload identity is ready.

Bootstrap Azure DevOps Self-Hosted Agent in Container

Azure DevOps offers a comprehensive guide on running an agent in a container, which you might find helpful, especially if you’re new to this concept. Since the guide is tailored for PAT usage, it requres a slight modification to accommodate AKS workload identity. Let’s begin with the Dockerfile. While the basic template is copied from the official guide, one important change has been made — the installation of the Azure CLI.

FROM ubuntu:22.04

RUN apt update && apt upgrade -y && \
    apt install -y ca-certificates curl apt-transport-https lsb-release gnupg jq libicu-dev git && \
    curl -sL https://aka.ms/InstallAzureCLIDeb | bash && \
    rm -rf /var/lib/apt/lists/*

RUN useradd -m agent

# Also can be "linux-arm", "linux-arm64".
ENV TARGETARCH="linux-x64"

WORKDIR /azp
COPY ./start.sh /azp/
RUN chmod +x /azp/start.sh && \
    chown -R agent:agent /azp && \
    chmod 755 /azp
USER agent

ENTRYPOINT [ "./start.sh" ]

Another file that needs modification is start.sh, the startup script for running the self-hosted agent. It requires three mandatory environment variables, as outlined below.

AZP Environment Variables for agent

The part we need to modify is the AZP_TOKEN section, originally designed for a PAT, which is clearly not what we want. We installed Azure CLI in the container image to get around this part. To address this, we installed Azure CLI in the container image. Rather than utilizing a PAT, we will dynamically generate an access token using AKS federated identity. Unfortunately, the Azure CLI currently doesn’t natively support federated identity, unlike Azure SDKs, so we will be using more explicit way. The two commands below generate an access token that expires every hour.

$ az login - federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" - service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID - allow-no-subscriptions
$ az account get-access-token - resource 499b84ac-1321–427f-aa17–267ca6975798 - query "accessToken" - output tsv

Below is an example of JWT token generated by commands above.

{
  "aud": "499b84ac-1321-427f-aa17-267ca6975798",
  "iss": "https://sts.windows.net/a6f0e660-***/",
  "iat": 1701703169,
  "nbf": 1701703169,
  "exp": 1701789869,
  "aio": "E2VgYNj4/***",
  "appid": "07e18ba0-***",
  "appidacr": "2",
  "idp": "https://sts.windows.net/a6f0e660-***/",
  "oid": "e4265182-***",
  "rh": "0.AbcAYObwpiR2uUq4LyEW_***",
  "sub": "e4265182-***",
  "tid": "a6f0e660-***",
  "uti": "***",
  "ver": "1.0"
}

What we need to do is to plug these two commands in the startup.sh to place the access token into AZP_TOKEN. I’ve added two lines below, watch the part with ## Generating access token to access Azure DevOps with federated identity

#!/bin/bash
set -e

if [ -z "${AZP_URL}" ]; then
  echo 1>&2 "error: missing AZP_URL environment variable"
  exit 1
fi

## Generating access token to access Azure DevOps with federated identity
az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --allow-no-subscriptions
AZP_TOKEN=$(az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query "accessToken" --output tsv)

if [ -z "${AZP_TOKEN_FILE}" ]; then
  if [ -z "${AZP_TOKEN}" ]; then
    echo 1>&2 "error: missing AZP_TOKEN environment variable"
    exit 1
  fi

  AZP_TOKEN_FILE="/azp/.token"
  echo -n "${AZP_TOKEN}" > "${AZP_TOKEN_FILE}"
fi

unset AZP_TOKEN

if [ -n "${AZP_WORK}" ]; then
  mkdir -p "${AZP_WORK}"
fi

cleanup() {
  trap "" EXIT

  if [ -e ./config.sh ]; then
    print_header "Cleanup. Removing Azure Pipelines agent..."

    # If the agent has some running jobs, the configuration removal process will fail.
    # So, give it some time to finish the job.
    while true; do
      ./config.sh remove --unattended --auth "PAT" --token $(cat "${AZP_TOKEN_FILE}") && break

      echo "Retrying in 30 seconds..."
      sleep 30
    done
  fi
}

print_header() {
  lightcyan="\033[1;36m"
  nocolor="\033[0m"
  echo -e "\n${lightcyan}$1${nocolor}\n"
}

# Let the agent ignore the token env variables
export VSO_AGENT_IGNORE="AZP_TOKEN,AZP_TOKEN_FILE"

print_header "1. Determining matching Azure Pipelines agent..."

AZP_AGENT_PACKAGES=$(curl -LsS \
    -u user:$(cat "${AZP_TOKEN_FILE}") \
    -H "Accept:application/json;" \
    "${AZP_URL}/_apis/distributedtask/packages/agent?platform=${TARGETARCH}&top=1")

AZP_AGENT_PACKAGE_LATEST_URL=$(echo "${AZP_AGENT_PACKAGES}" | jq -r ".value[0].downloadUrl")

if [ -z "${AZP_AGENT_PACKAGE_LATEST_URL}" -o "${AZP_AGENT_PACKAGE_LATEST_URL}" == "null" ]; then
  echo 1>&2 "error: could not determine a matching Azure Pipelines agent"
  echo 1>&2 "check that account "${AZP_URL}" is correct and the token is valid for that account"
  exit 1
fi

print_header "2. Downloading and extracting Azure Pipelines agent..."

curl -LsS "${AZP_AGENT_PACKAGE_LATEST_URL}" | tar -xz & wait $!

source ./env.sh

trap "cleanup; exit 0" EXIT
trap "cleanup; exit 130" INT
trap "cleanup; exit 143" TERM

print_header "3. Configuring Azure Pipelines agent..."

./config.sh --unattended \
  --agent "${AZP_AGENT_NAME:-$(hostname)}" \
  --url "${AZP_URL}" \
  --auth "PAT" \
  --token $(cat "${AZP_TOKEN_FILE}") \
  --pool "${AZP_POOL:-Default}" \
  --work "${AZP_WORK:-_work}" \
  --replace \
  --acceptTeeEula & wait $!

print_header "4. Running Azure Pipelines agent..."

chmod +x ./run.sh

# To be aware of TERM and INT signals call ./run.sh
# Running it with the --once flag at the end will shut down the agent after the build is executed
./run.sh "$@" & wait $!

Two changes are made, and it’s time to build it. Build the image and push it to your own container registry.

$ docker build -t eggboy/ado-agent:3.0.1 --file ./Dockerfile .

I will leave Deployment yaml here to run self-hosted agents on Kubernetes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ado-agent-fi
  labels:
    app: ado-agent-fi
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ado-agent-fi
  template:
    metadata:
      name: ado-agent-fi
      labels:
        app: ado-agent-fi
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: ado-agent-sa
      containers:
        - name: ado-agent-fi
          image: eggboy/ado-agent-fi:3.0.0
          imagePullPolicy: Always
          env:
            - name: AZP_URL
              value: https://dev.azure.com/***
            - name: AZP_POOL
              value: aks-federated-identity
            - name: AZP_AGENT_NAME
              value: aks-agent
      restartPolicy: Always

Set up Managed Identity to Azure DevOps Organization

The final step is to integrate managed identities into your Azure DevOps organization to provide access to organizational resources. Navigate to Organization settings and search for the name of the previously created managed identity.

Adding MI to Azure DevOps

This step grants access to the Managed Identity, allowing it to access your ADO Organization. Additionally, you need to add permissions to the managed identity for running pipelines. In my case, I’ve assigned Administrator access to the managed identity.

Self-hosted Agent is ready to be used at this point. Let’s run the sample pipeline to validate our setup.

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- main

pool:
  name: "aks-federated-identity"

steps:
- script: |
    az account show
  displayName: 'Show account info'

Pipeline runs

Wrapping Up

Running self-hosted agent without PAT is one major step to get rid of the PAT usage in Azure DevOps Pipeline architecture. My follow-up article is to setup DevOps pipeline with Federated Identity that will use JWT token for Azure authentication, and it will be very similar to the GitHub Action approach. Stay tuned for the next article.

If you liked my article, please leave a few claps or start following me. You can get notified whenever I publish something new. Let’s stay connected on Linkedin, too! Thank you so much for reading!