Azure DevOps Build Pipeline can't get secrets from Key Vault when secured with vnet and firewall

Question

Can't get secrets from Key Vault when it's secured with vnet and firewall.

I would like to use secrets stored in key vault from DevOps Build Pipeline task and I would like to follow security best practice and defense in depth. As security best practice, I want key vault to be accessible from selected virtual networks, selected azure services and from trusted internet ip's. Of course, I would use a service principal and appropriate permissions (list/get).

Unfortunately, Azure DevOps is not one of the trusted service. So, my alternative is to white-list the DevOps IPs. I found out my DevOps is in US East 2 region and I downloaded Azure Datacenter IPs (filtered with US East2). There are about 285 IP's in US East 2. Key Vault firewall has a limit on how many firewall rules you can add and it's 127! So, I am out of luck!

At the moment, I can get secrets from key vault at build pipeline only if I allow all networks! Yea, I still have to authenticate to get the secrets but I lost on defense in depth. I really need to lockdown the key vault to trusted networks but I can't. Why? I can't add more than 127 firewall rules (to cover the region) and DevOps is not one of the trusted azure services!

Answer 1

I thought I'd add to this with a twist on the solution that Prodip provided. This one relies on the fact that when you request a secret, the az client is kind enough to tell you what your client IP address is, i.e:

az keyvault secret show -n "a-known-client-secret" --vault-name "$keyVaultName"

Attempting to get value for known secret from key vault: '******'
ERROR: Client address is not authorized and caller is not a trusted service.
Client address: 1.1.1.1
Caller: appid=***;oid=****;iss=https://sts.windows.net/***/
Vault: ******;location=******

So here's my bash script (whitelist-agent-for-key-vault.sh):

#!/usr/bin/env bash

## By default the Azure DevOps IP addresses are NOT whitelisted for key vault access. So even if the service principal has access, you won't get past the firewall.
## The solution is to temporarily add the build agent IP address to the key vault firewall, and remove it when the pipeline is complete. 

if [[ $(uname -s) == "Linux" ]]; then
    azcmd="az"
else
    # If we're in a bash shell on Windows, az commands don't work, but we can call the az.cmd batch file directly from git Bash if we can find it...
    azcmd=$(where az.cmd)
fi

# Are we removing rather than setting?
if [[ $1 == "-r" ]]; then
    if [[ -z "$3" ]]; then
        echo "Build agent IP address is empty, no whitelist entry to remove from key vault: '$2'"
    else
        echo "Removing key vault '$2' network rule for DevOps build agent IP address: '$3'"

        # Remember to specify CIDR /32 for removal
        "$azcmd" keyvault network-rule remove -n $2 --ip-address $3/32
    fi
    exit 0
fi

keyVaultName=$1

########################################################################
##### This is the known secret which we request from the key vault #####
########################################################################

knownSecret="<My known secret>"

echo "Attempting to get value for known secret from key vault: '$keyVaultName'"

# Attempt to show secret - if it doesn't work, we are echoed our IP address on stderror, so capture it
secretOutput=$("$azcmd" keyvault secret show -n "$knownSecret" --vault-name "$keyVaultName" 2>&1)
buildAgentIpAddress=$(echo $secretOutput | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b")

set -euo pipefail

if [[ ! -z "$buildAgentIpAddress" ]]; then
    # Temporarily whitelist Azure DevOps IP for key vault access.
    # Note use of /32 for CIDR = 1 IP address. If we omit this Azure adds it anyway and fails to match on the IP when attempting removal.
    echo "Azure DevOps IP address '$buildAgentIpAddress' is blocked. Attempting to whitelist..."
    "$azcmd" keyvault network-rule add -n $keyVaultName --ip-address $buildAgentIpAddress/32

    # Capture the IP address as an ADO variable, so that this can be undone in a later step
    echo "##vso[task.setvariable variable=buildAgentIpAddress]$buildAgentIpAddress"
else
    # We didn't find the IP address - are we already whitelisted?
    secretValue=$(echo $secretOutput | grep -o "value")

    if [[ -z "$secretValue" ]]; then
        echo "Unexpected response from key vault whitelist request, json attribute 'value' not found. Unable to whitelist build agent - response was: '$secretOutput'"
        exit 1
    fi
fi

Here's how I add the IP to the white list:

  # Add agent IP to key vault white list
  - task: AzureCLI@2
    displayName: Add Azure DevOps build agent IP to key vault white list
    inputs:
      azureSubscription: ${{ parameters.azureSubscription }}
      scriptType: bash
      scriptLocation: scriptPath
      scriptPath: $(Pipeline.Workspace)/server-build-tools/drop/build-scripts/whitelist-agent-for-key-vault.sh
      arguments: '$(keyVaultName)'

Here's how I remove the IP from the white list

      - task: AzureCLI@2
        displayName: Remove Azure DevOps build agent IP from key vault white list
        condition: always()
        inputs:
          azureSubscription: ${{ parameters.azureSubscription }}
          scriptType: bash
          scriptLocation: scriptPath
          scriptPath: $(Pipeline.Workspace)/server-build-tools/drop/build-scripts/whitelist-agent-for-key-vault.sh
          arguments: '-r "$(keyVaultName)" "$(buildAgentIpAddress)"'

Caveats:

• This relies on the Azure DevOps service principal having been granted read access to the key vault secrets
• Replace the knownSecret value with the name of your known secret

Bonus:

This works using Azure CLI and has been tested on Azure DevOps for Linux and Windows build agents running under Git bash for the latter. Normally if you try running 'az' commands in Git Bash, you just get 'Command not found'. I wanted a solution that would work on both, since I need to share code because of the Linux / Windows build requirement.

Answer 2

You can add a step in the build definition to whitelist the agent IP address, then remove it from the whitelist at the end of the build. This is not a solution but a workaround until Azure product team adds Azure DevOps as a trusted service. Thanks to @DanielMann for providing the idea.

The solution is simple but I was not going to trust ipify.org as REST API endpoint to get my build agent’s ip address. Instead, I created my own (and trusted) service at Azure Function- GetClientIP. DevOps is not my day job and I was having hard time to figure out how to assign and use user defined variables and pass them on to next step/task/stage in pipeline! Microsoft documentation on variables usages was not helping me enough but I figured it out after lot of unsuccessful runs!

See the complete solution at my blog- Azure DevOps Build Pipeline- use keys and secrets from Key Vault.