Hiredis timeout with large number of concurrent requests

Question

I'm using a redis integration plugin for Varnish called libvmod-redis. I'm seeing an issue where if I get a large number of concurrent requests, around 350, redis starts timing out and I eventually get a segfault in Varnish.

I get these errors:

varnishd[27892]: Child (27893) said redis error (connect): Connection timed out
varnishd[27892]: Child (27893) said redis error (command): err=1 errstr=Connection timed out
varnishd[19528]: Child (19529) said redis error (command): err=1 errstr=select(2): Invalid argument
varnishd[19528]: Child (19529) said redis error (command): err=1 errstr=Connection timed out
varnishd[19528]: last message repeated 9 times
varnishd[19528]: Child (19529) said redis error (command): err=1 errstr=select(2): Invalid argument
varnishd[19528]: Child (19529) said redis error (connect): fcntl(F_GETFL): Bad file descriptor
varnishd[19528]: Child (19529) said redis error (command): err=1 errstr=fcntl(F_GETFL): Bad file descriptor
kernel: [282284.005658] varnishd[19727] general protection ip:7f1f9dea1427 sp:7f1f4123c120 error:0 in libhiredis.so.0.10[7f1f9de9f000+9000]

My timeout is 1 second, and I'm using an ElastiCache node for Redis. I'm wondering what exactly could be failing here. I'm not an expert in C, so I feel like I'm missing something.

Answer 1

It would be interesting to see the code, especially around the select. Since you have a lot of concurrent requests you better check the validity of your file descriptors: maybe on of them is closed during the process...

Answer 2

You may try the following:

• To start, try with redis-cli --latency if the problem is the Redis server. If redis-cli has no latency issues from its point of view, the problem is the client.
• If the problem is the server, you may want to follow this guide: http://redis.io/topics/latency