Redis connection/buffer-size limit exceeded

Question

While stress testing our application server we have got the following exception from Redis:

ServiceStack.Redis.RedisException: could not connect to redis Instance at redis-host:6379 ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full redis-host:6379 at System.Net.Sockets.Socket.Connect(IPAddress[] addresses, Int32 port) at System.Net.Sockets.Socket.Connect(String host, Int32 port) at ServiceStack.Redis.RedisNativeClient.Connect() --- End of inner exception stack trace --- at ServiceStack.Redis.RedisNativeClient.Connect() at ServiceStack.Redis.RedisNativeClient.AssertConnectedSocket() at ServiceStack.Redis.RedisNativeClient.SendCommand(Byte[][] cmdWithBinaryArgs) at ServiceStack.Redis.RedisNativeClient.SendExpectData(Byte[][] cmdWithBinaryArgs) at ServiceStack.Redis.RedisClient.GetValueFromHash(String hashId, String key) at ServiceStack.Redis.Generic.RedisTypedClient1.GetValueFromHash[TKey](IRedisHash2 hash, TKey key)

It seems that there are connection limit exceeds on redis host port. Any idea how to increase this threshold through Redis.conf OR server configuration? We have hosted the Redis instance over Ubuntu server.

Answer 1

I was able to duplicate the same issue of buffer size limit exceeded using ServiceStack. The code to do the stress testing is here - run 20 instances of the application for at least 20 minutes. https://github.com/ServiceStack/ServiceStack.Redis/commit/b01582f9c873f375794c04d46aad400590ca5bf3

The first error you may see is "Could not connect to redis instance" as described by Redis unable to connect in busy load, but if you expand the inner exception you see "An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full"

My problem occured on Window7, but not Window Server 2008 rc. So I begin to look at if it was an OS problem. After emailing Demis at ServiceStack, it was concluded that ServiceStack was closing the sockets correctly. Looking at the OS, the problem was fixed with setting TcpTimeWaitDelay and MaxUserPort.

More references. TcpTimeWaitDelay to 45 seconds

and MaxUserPort http://mashijie.blogspot.com/2009/05/change-default-setting-of-tcp-ports.html

I adjusted the port range to 1025-64511