Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

StackExchange.Redis with Azure Redis is unusably slow or throws timeout errors

I'm moving all of my existing Azure In-Role cache use to Redis and decided to use the Azure Redis preview along with the StackExchange.Redis library (https://github.com/StackExchange/StackExchange.Redis). I wrote all the code for it without much problem, but when running it is absolutely unusably slow and constantly throws timeout errors (my timeout period is set to 15 seconds).

Here is the relevant code for how I am setting up the Redis connection and using it for simple operations:

    private static ConnectionMultiplexer _cacheService;     private static IDatabase _database;     private static object _lock = new object();      private void Initialize()     {         if (_cacheService == null)         {             lock (_lock)             {                 if (_cacheService == null)                 {                     var options = new ConfigurationOptions();                     options.EndPoints.Add("{my url}", 6380);                     options.Ssl = true;                     options.Password = "my password";                     // needed for FLUSHDB command                     options.AllowAdmin = true;                      // necessary?                     options.KeepAlive = 30;                     options.ConnectTimeout = 15000;                     options.SyncTimeout = 15000;                      int database = 0;                      _cacheService = ConnectionMultiplexer.Connect(options);                     _database = _cacheService.GetDatabase(database);                 }             }         }      }      public void Set(string key, object data, TimeSpan? expiry = null)     {         if (_database != null)         {             _database.Set(key, data, expiry: expiry);         }     }      public object Get(string key)     {         if (_database != null)         {             return _database.Get(key);         }         return null;     } 

Performing very simple commands like Get and Set often time out or take 5-10 seconds to complete. Seems like it kind of negates the whole purpose of using it as a cache if it's WAY slower than actually fetching the real data from my database :)

Am I doing anything obviously incorrect?

Edit: here are some stats that I pulled from the server (using Redis Desktop Manager) in case that sheds some light on anything.

Server redis_version:2.8.12 redis_mode:standalone os:Windows   arch_bits:64 multiplexing_api:winsock_IOCP gcc_version:0.0.0 process_id:2876  tcp_port:6379 uptime_in_seconds:109909 uptime_in_days:1 hz:10 lru_clock:16072421 config_file:C:\Resources\directory\xxxx.Kernel.localStore\1\redis_2092_port6379.conf  Clients connected_clients:5 client_longest_output_list:0 client_biggest_input_buf:0 client_total_writes_outstanding:0 client_total_sent_bytes_outstanding:0 blocked_clients:0  Memory used_memory:4256488 used_memory_human:4.06M used_memory_rss:67108864 used_memory_rss_human:64.00M used_memory_peak:5469760 used_memory_peak_human:5.22M used_memory_lua:33792 mem_fragmentation_ratio:15.77 mem_allocator:dlmalloc-2.8  Persistence loading:0 rdb_changes_since_last_save:72465 rdb_bgsave_in_progress:0 rdb_last_save_time:1408471440 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok  Stats total_connections_received:25266 total_commands_processed:123389 instantaneous_ops_per_sec:10 bytes_received_per_sec:275 bytes_sent_per_sec:65 bytes_received_per_sec_human: 

Edit 2: Here are the extension methods I'm using for Get/Set -- they are very simple methods that just turn an object into JSON and call StringSet.

    public static object Get(this IDatabase cache, string key)     {         return DeserializeJson<object>(cache.StringGet(key));     }      public static void Set(this IDatabase cache, string key, object value, TimeSpan? expiry = null)     {         cache.StringSet(key, SerializeJson(value), expiry: expiry);     } 

Edit 3: here are a couple example error messages:

    A first chance exception of type 'System.TimeoutException' occurred in StackExchange.Redis.dll     Timeout performing GET MyCachedList, inst: 11, queue: 1, qu=1, qs=0, qc=0, wr=0/1, in=0/0      A first chance exception of type 'System.TimeoutException' occurred in StackExchange.Redis.dll     Timeout performing GET MyCachedList, inst: 1, queue: 97, qu=0, qs=97, qc=0, wr=0/0, in=3568/0 
like image 319
user2719100 Avatar asked Aug 21 '14 00:08

user2719100


People also ask

Does Redis have a timeout?

Redis client uses a single TCP connection and can only read one response at a time. Even when a first operation times out, it does not stop the data being sent to/from the server. Because of this, it blocks other requests and causes timeouts.

What is the difference between Redis and Azure cache for Redis?

Azure Cache for Redis provides an in-memory data store based on the Redis software. Redis improves the performance and scalability of an application that uses backend data stores heavily.

What is the default timeout for Redis cache?

Azure Cache for Redis service configures a 10 minute idle server timeout for connections, independently of the client library used. This is defined on Server side and cannot be changed.


2 Answers

Here is the recommended pattern, from the Azure Redis Cache documentation:

private static Lazy<ConnectionMultiplexer> lazyConnection = new Lazy<ConnectionMultiplexer>(() => {     return ConnectionMultiplexer.Connect("mycache.redis.cache.windows.net,abortConnect=false,ssl=true,password=..."); });  public static ConnectionMultiplexer Connection {     get {         return lazyConnection.Value;     } } 

A few important points:

  • It uses Lazy<T> to handle thread-safe initialization
  • It sets "abortConnect=false", which means if the initial connect attempt fails, the ConnectionMultiplexer will silently retry in the background rather than throw an exception.
  • It does not check the IsConnected property, since ConnectionMultiplexer will automatically retry in the background if the connection is dropped.
like image 55
Mike Harder Avatar answered Sep 19 '22 16:09

Mike Harder


I was having similar issues. Redis cache was unusually slow but was definitely caching. In some cases, it took 20-40 seconds to load a page.

I realized that the cache server was in a different location than the site's. I updated the cache server to live in the same location as the website and now everything works as expected.

That same page now loads in 4-6 seconds.

Good luck to anyone else who's having these issues.

like image 41
christo8989 Avatar answered Sep 19 '22 16:09

christo8989