Azure Key Vault secret access intermittently failing to connect with socket exception

Question

I have an MVC 5 web application running on .NET 4.7.2 and hosted in an Azure AppService, that uses Azure Key Vault to hold secrets. The project uses the Microsoft.Azure.KeyVault 3.0.3 NuGet package and the secrets are accessed using the KeyVaultClient and .GetSecretAsync(). All resources are located in the same Azure region.

For the most part this works very well, and for about 90% of the time it returns the secret in milliseconds.

But every now and then the call to access the Key Vault fails. This doesn't manifest itself as an exception thrown by the SDK, but the web app hangs. Eventually - and normally in around 1 minute but sometimes longer - the secret is returned and all is fine again. This is because the SDK uses a retry pattern, which will keep trying to get the secret.

Looking at Application Insights for the AppService I can see that the GET request generated by the SDK gets an HTTP 500 response from the Key Vault and a SocketException is thrown, with a result code of ConnectFailure.

The exception is:

Looking at the telemetry and stepping through the code there is no element of commonality or obvious cause. It seems to be entirely random.

The bottom line is the Azure hosted AppService sometimes cannot connect to an Azure hosted Key Vault in the same datacentre, using the latest framework and SDK version.

Has anyone else seen this or have any idea? I've searched around and found a few instances of people experiencing the same issue, but nobody has a cause or solution.

EDIT (1): I have now tried spinning up a new Key Vault in a different region entirely, and the problem remains exactly the same.

Answer 1

We experienced the same behavior on our project, where KeyVault would be fast and reliable most of the time, and then intermittently stop responding or take a very long time to return once in a while with no obvious reason to explain why. This occurred in all tiers of our application, from the API, to Azure Functions, to command line tools.

Eventually, we had to work around this by caching secrets in memory to avoid hitting the KeyVault too often, where our AppSettings class would cache these internally. In addition to this, we also configured our DI container to treat this class as a singleton.

Here is a very simplified example:

public class MyAppSettings : IAppSettings
{
    private readonly ObjectCache _cache = MemoryCache.Default;
    private readonly object _lock = new Object();
    private KeyValueClient _kvClient;

    public string MySecretValue => GetSecret("MySecretValue");

    private KeyValueClient GetKeyVaultClient()
    {
        // Initialize _kvClient if required

        return _kvClient;
    }

    private string GetSecret(string name)
    {
        lock (_lock)
        {
            if (_cache.Contains(key))
                return (string) _cache.Get(key);

            // Sanitize name if required, remove reserved chars

            // Construct path
            var path = "...";

            // Get value from KV

            var kvClient = GetKeyVaultClient();
            Task<SecretBundle> task = Task.Run(async() => await kvClient.GetSecretAsync(path));

            var value = task.Result;

            // Cache it
            _cache.Set(name, value, DateTime.UtcNow.AddHours(1));

            return value;
        }
    }
}

This isn't production ready - you'll need to modify this and implement the GetKeyVaultClient method to actually return your KeyVaultClient object, and also the GetSecret method should sanitize the key name being retrieved.

In our DI registry, we had this setup to use a singleton like this:

For<IAppSettings>().Use<MyAppSettings>().Singleton();

These two changes seemed to work well for us, and we haven't had any issues with this for a while now.