VSTS Build Pipeline: Test fails connecting to Azure Key Vault

Question

I am trying to use VSTS (now Azure DevOps) to do a CI/CD pipeline. For my build pipeline, I have a very basic setup involving doing a restore, build, test, and publish steps.

For my test step, I have it setup to run two test projects - one unit test project and one integration test project. I have my Key Vault access policy setup to provide access to both myself and Azure Devops. When I run my tests locally using visual studio, as I am logged into the same account which has access to azure key vault, I can run the tests without any errors.

My application is configured to access key vault using below setup:

 public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration((ctx, builder) =>
            {
                var keyVaultEndpoint = GetKeyVaultEndpoint();

                if (!string.IsNullOrEmpty(keyVaultEndpoint))
                {
                    var azureServiceTokenProvider = new AzureServiceTokenProvider();
                    var keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
                    builder.AddAzureKeyVault(keyVaultEndpoint, keyVaultClient, new DefaultKeyVaultSecretManager());
                }
            }
        )
            .UseStartup<Startup>();

When I run the build pipeline, I am using a Hosted VS2017 instance to build my project. Everything is running except the integration tests which try and access the key vault fail. I am using the following packages:

• Microsoft.Azure.Services.AppAuthentication - makes it easy to fetch access tokens for Service-to-Azure-Service authentication scenarios.
• Microsoft.Azure.KeyVault - contains methods for interacting with Key Vault.
• Microsoft.Extensions.Configuration.AzureKeyVault - contains
  IConfiguration extensions for Azure Key Vault

I followed this tutorial https://docs.microsoft.com/en-us/azure/key-vault/tutorial-web-application-keyvault to setup the key vault and integrate it into my app.

I am merely trying to get my build to work by making sure both the unit and integration tests pass. I am not deploying it to an app service yet. The unit tests run without any issues as I am mocking the various services. My integration test is failing with below error messages. How do I get my test access to the key vault? Do I need to add any special access policies to my key vault for the hosted VS2017 build? Not sure what to do as I don't see anything that stands out.

Below is the stack trace for the error:

    2018-10-16T00:37:04.6202055Z Test run for D:\a\1\s\SGIntegrationTests\bin\Release\netcoreapp2.1\SGIntegrationTests.dll(.NETCoreApp,Version=v2.1)
    2018-10-16T00:37:05.3640674Z Microsoft (R) Test Execution Command Line Tool Version 15.8.0
    2018-10-16T00:37:05.3641588Z Copyright (c) Microsoft Corporation.  All rights reserved.
    2018-10-16T00:37:05.3641723Z 
    2018-10-16T00:37:06.8873531Z Starting test execution, please wait...
    2018-10-16T00:37:51.9955035Z [xUnit.net 00:00:40.80]     SGIntegrationTests.HomeControllerShould.IndexContentTypeIsTextHtml [FAIL]
    2018-10-16T00:37:52.0883568Z Failed   SGIntegrationTests.HomeControllerShould.IndexContentTypeIsTextHtml
    2018-10-16T00:37:52.0884088Z Error Message:
    2018-10-16T00:37:52.0884378Z  Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProviderException : Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/63cd8468-5bc3-4c0a-a6f8-1e314d696937. Exception Message: Tried the following 3 methods to get an access token, but none of them worked.
    2018-10-16T00:37:52.0884737Z Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/63cd8468-5bc3-4c0a-a6f8-1e314d696937. Exception Message: Tried to get token using Managed Service Identity. Access token could not be acquired. MSI ResponseCode: BadRequest, Response: {"error":"invalid_request","error_description":"Identity not found"}
    2018-10-16T00:37:52.0884899Z Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/63cd8468-5bc3-4c0a-a6f8-1e314d696937. Exception Message: Tried to get token using Visual Studio. Access token could not be acquired. Visual Studio Token provider file not found at "C:\Users\VssAdministrator\AppData\Local\.IdentityService\AzureServiceAuth\tokenprovider.json"
    2018-10-16T00:37:52.0885142Z Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/63cd8468-5bc3-4c0a-a6f8-1e314d696937. Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. Process took too long to return the token.
    2018-10-16T00:37:52.0885221Z 
    2018-10-16T00:37:52.0885284Z Stack Trace:
    2018-10-16T00:37:52.0885349Z    at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.GetAccessTokenAsyncImpl(String authority, String resource, String scope)
    2018-10-16T00:37:52.0885428Z    at Microsoft.Azure.KeyVault.KeyVaultCredential.PostAuthenticate(HttpResponseMessage response)
    2018-10-16T00:37:52.0885502Z    at Microsoft.Azure.KeyVault.KeyVaultCredential.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    2018-10-16T00:37:52.0886831Z    at Microsoft.Azure.KeyVault.KeyVaultClient.GetSecretsWithHttpMessagesAsync(String vaultBaseUrl, Nullable`1 maxresults, Dictionary`2 customHeaders, CancellationToken cancellationToken)
    2018-10-16T00:37:52.0886887Z    at Microsoft.Azure.KeyVault.KeyVaultClientExtensions.GetSecretsAsync(IKeyVaultClient operations, String vaultBaseUrl, Nullable`1 maxresults, CancellationToken cancellationToken)
    2018-10-16T00:37:52.0886935Z    at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.LoadAsync()
    2018-10-16T00:37:52.0887000Z    at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.Load()
    2018-10-16T00:37:52.0887045Z    at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
    2018-10-16T00:37:52.0887090Z    at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
    2018-10-16T00:37:52.0887269Z    at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
    2018-10-16T00:37:52.0887324Z    at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
    2018-10-16T00:37:52.0887371Z    at Microsoft.AspNetCore.TestHost.TestServer..ctor(IWebHostBuilder builder, IFeatureCollection featureCollection)
    2018-10-16T00:37:52.0887433Z    at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateServer(IWebHostBuilder builder)
    2018-10-16T00:37:52.0887477Z    at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.EnsureServer()
    2018-10-16T00:37:52.0887525Z    at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.CreateDefaultClient(DelegatingHandler[] handlers)

---

Update

I have found only 1 related post to this issue: https://social.msdn.microsoft.com/Forums/en-US/0bac778a-283a-4be1-bc75-605e776adac0/managed-service-identity-issue?forum=windowsazurewebsitespreview. But the post is related to deploying an application into an azure slot. I am merely trying to build my application in a build pipeline.

I am still trying to solve this issue and am not sure what the best way to provide the required access is.

---

Update 2

I have still not found a solution for this. I am lost on how to get my pipeline to run my test without issues. I saw that the release pipeline you have the options of running tests too. But these seem to take .dll files and my build pipeline drop file only has the web app (I don't see any of the test projects published drop file). Not sure if that is even a possibility.

---

Update 3

I managed to get it to work by using the last option provided here: https://docs.microsoft.com/en-us/azure/key-vault/service-to-service-authentication#connection-string-support

I tried the other ways of using a certificate but anytime {CurrentUser} is provided in a connection string, the build pipeline fails. It works on my local machine but not in the build pipeline.

To get it to work, I had to do three things:

• Log in to Azure. Setup a new app registration in Azure AD
• In your new AD app registration, create a new client secret

• Provide your new AD App access to your key vault. Go into your key vault access policies and add the app that you created in your AD with read access to your secrets.

• Modified my call to AzureServiceTokenProvier() in my Program.cs file as follows:

   var azureServiceTokenProvider = new AzureServiceTokenProvider("connectionString={your key vault endpoint};RunAs=App;AppId={your app id that you setup in Azure AD};TenantId={your azure subscription};AppKey={your client secret key}")
  

Note that your client secret has to be formatted correctly. The app registrations (preview) generates a random secret key. Sometimes this key does not work in the connection string (throws an error as incorrectly formatted). Either try generating your own key in the non-preview version of app registration or generate a new key and try again.

After that I was able to run my integration test in my build pipeline successfully and create a release to my web app in Azure. I'm not satisfied with this approach because although it works, its exposing a secret value in the code itself. Manages service identity does not need to be turned on due to above approach. I feel that this is extremely bad in that regard.

There has to be a better way than this. One option is not to run the integration test in the build pipeline. Not sure if this is the correct approach. I'm still hoping someone will be able to provide a better approach to this or explain if my approach is okay to use.

Answer 1

Use the Azure CLI pipeline task to run integration tests that need KeyVault secrets successfully, without exposing any secrets in source control:

1. Create a Service Principal service connection in your Azure DevOps project.

2. Give the principal Get and List permissions to the Vault in Azure.

3. Run your integration tests inside an Azure CLI task:

   - task: AzureCLI@1
     inputs:
       azureSubscription: 'Your Service Connection Name'
       scriptLocation: 'inlineScript'
       inlineScript: 'dotnet test --configuration $(buildConfiguration) --logger trx'
   

   This works because the tests will run in the context of azure cli, which is where AzureServiceTokenProvider tries fetching a token from before it fails. Azure CLI handles the authentication and cleans up when the task is done.