No connection is available to service this operation: when using Azure Redis Cache

Question

I have the following code which I use to get information from the cache. I dont know if maybe my app is opening too many connections or just this error is due to a transient failure on azure redis cache.

This is the stack trace

[RedisConnectionException: No connection is available to service this operation: GET UserProfileInformation|[email protected]] StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl(Message message, ResultProcessor1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\ConnectionMultiplexer.cs:1922 StackExchange.Redis.RedisBase.ExecuteSync(Message message, ResultProcessor1 processor, ServerEndPoint server) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:80 StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisDatabase.cs:1431 xx.Utils.SampleStackExchangeRedisExtensions.Get(IDatabase cache, String key) in C:\Proyectos\xx\xx\Utils\SampleStackExchangeRedisExtensions.cs:20
xx.Cache.UserProfile.GetUserProfile(String identityname) in C:\Proyectos\xx\xx\Cache\UserProfile.cs:22
x.Controllers.UserProfileController.GetPropertiesForUser() in C:\Proyectos\xx\xx\Controllers\UserProfileController.cs:16
lambda_method(Closure , ControllerBase , Object[] ) +61
System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters) +14

And this is the code

   public static Models.UserProfile GetUserProfile(string identityname)
        {
            /// It needs to be cached for every user because every user can have different modules enabled.

            var cachekeyname = "UserProfileInformation|" + identityname;
            IDatabase cache = CacheConnectionHelper.Connection.GetDatabase();
            Models.UserProfile userProfile = new Models.UserProfile();
            object obj = cache.Get(cachekeyname);
            string userProfileString;
            if (obj != null)
            {
                //get string from cache
                userProfileString = obj.ToString();

                //conver string to our object
                userProfile = JsonConvert.DeserializeObject<Models.UserProfile>(userProfileString);
                return userProfile;
            }
            else
            {
                #region Get User Profile from AD
                Uri serviceRoot = new Uri(SettingsHelper.AzureAdGraphApiEndPoint);
                var token = AppToken.GetAppToken();

                ActiveDirectoryClient adClient = new ActiveDirectoryClient(
                 serviceRoot,
                 async () => await AppToken.GetAppTokenAsync());

                string userObjectID = ClaimsPrincipal.Current.FindFirst("http://schemas.microsoft.com/identity/claims/objectidentifier").Value;

                Microsoft.Azure.ActiveDirectory.GraphClient.Application app = (Microsoft.Azure.ActiveDirectory.GraphClient.Application)adClient.Applications.Where(
                    a => a.AppId == SettingsHelper.ClientId).ExecuteSingleAsync().Result;
                if (app == null)
                {
                    throw new ApplicationException("Unable to get a reference to application in Azure AD.");
                }

                string requestUrl = string.Format("https://graph.windows.net/{0}/users/{1}?api-version=1.5", SettingsHelper.Tenant, identityname);
                HttpClient hc = new HttpClient();
                hc.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token);
                HttpResponseMessage hrm = hc.GetAsync(new Uri(requestUrl)).Result;

                if (hrm.IsSuccessStatusCode)
                {
                    Models.UserProfile currentUserProfile = JsonConvert.DeserializeObject<Models.UserProfile>(hrm.Content.ReadAsStringAsync().Result);

                    //convert object to json string
                    userProfileString = JsonConvert.SerializeObject(currentUserProfile);

                    cache.Set(cachekeyname, userProfileString, TimeSpan.FromMinutes(SettingsHelper.CacheUserProfileMinutes));
                    return currentUserProfile;
                }
                else
                {
                    return null;
                }
                #endregion
            }
        }




public static class SampleStackExchangeRedisExtensions
    {
        public static T Get<T>(this IDatabase cache, string key)
        {
            return Deserialize<T>(cache.StringGet(key));
        }

        public static object Get(this IDatabase cache, string key)
        {
            return Deserialize<object>(cache.StringGet(key));
        }

        public static void Set(this IDatabase cache, string key, object value, TimeSpan expiration)
        {
            cache.StringSet(key, Serialize(value), expiration);
        }

        static byte[] Serialize(object o)
        {
            if (o == null)
            {
                return null;
            }
            BinaryFormatter binaryFormatter = new BinaryFormatter();
            using (MemoryStream memoryStream = new MemoryStream())
            {
                binaryFormatter.Serialize(memoryStream, o);
                byte[] objectDataAsStream = memoryStream.ToArray();
                return objectDataAsStream;
            }
        }

        static T Deserialize<T>(byte[] stream)
        {
            BinaryFormatter binaryFormatter = new BinaryFormatter();
            if (stream == null)
                return default(T);

            using (MemoryStream memoryStream = new MemoryStream(stream))
            {
                T result = (T)binaryFormatter.Deserialize(memoryStream);
                return result;
            }
        }

Questions are: 1. How can I control a connection exception like the one shown, so that the user doesnt get the error and instead it goes to the DB if redis is unavailable? 2. Is there anyway to retry with transient fault handling for azure redis cache?

Answer 1

I believe these are transient errors. I have seen many of these in my application logs before I implemented simple retry logic. I also had quite a few timeouts. Very simple retry logic, plus adding syncTimeout=3000 to redis connection string resolved all these for me.

public object Get(string key)
{
    return Deserialize(Cache.StringGet(key));
}

public object GetWithRetry(string key, int wait, int retryCount)
{
    int i = 0;
    do
    {
        try
        {
            return Get(key);
        }
        catch (Exception)
        {
            if (i < retryCount + 1)
            {
                Thread.Sleep(wait);
                i++;
            }
            else throw;
        }
    }
    while (i < retryCount + 1);
    return null;
}

Answer 2

As Rouen suggests, these are probably transient connection errors. Here's a full async example using Polly to handle the retries.

        var value = await Policy 
            .Handle<RedisConnectionException>() // Possible network issue 
            .WaitAndRetryAsync(3, i => TimeSpan.FromSeconds(3)) // retry 3 times, with a 3 second delay, before giving up
            .ExecuteAsync(async () => {
                return await cache.StringGetAsync(key);
            });