Redis request latency

Question

I'm using redis (2.6.8) with php-fpm and phpredis driver and have some trouble with redis latency issues. Under certain load first request to redis from our application takes about 1-1.5s and redis-cli --latency shows the same latency.

I've already checked the latency guide.

• We use redis on the same host with Unix sockets
• slowlog has no entries longer 5ms
• we don't use AOF
• redis takes about 3.5Gb memory of 16Gb available (i suppose it's not too much)
• our system is not swapping
• there is no other process doing disk I/O

I'm using persistent connections and amount of connected client is varying from 5 to 25 (sometimes strikes to 60-80).

Here is the graph.

It looks like problems starts when there are 20 or more simultaneously connected clients.

Can you help me to figure out where is the problem?

Update

I investigated the problem and it seemed like redis did not have enough processor time for some reason to operate properly.

I thoroughly checked communication between php-fpm and redis with the help of network sniffer. Redis received request over tcp but sent the answer back only after one and a half seconds. It obviously signified that the problem is inside redis, that it cannot process so many requests in the given conditions (possibly processor starvation as the processor was only 50% loaded for the whole system).

The problem was resolved by moving redis to other server that was nearly idle. I suppose that we should have played with linux scheduler to make it work on the same server, but have not done it yet.

Answer 1

Bear in mind that Redis is single-threaded. If the operations that you're doing err on the processor-intensive side, your requests could be blocking on each other. For instance, if you're doing HVALS against hashes with very large values, you're going to make all of your clients wait while you pull out all that data and copy it to the output buffer.

Part of what you need to do here (regardless if this is the issue) is to look at all of the commands that you're using and determine the complexity of each command. If you're doing a bunch of O(N) commands against very large amounts of data, it's not impossible that you're simply doing too much stuff at a time.

TL;DR Nobody on here can debug this issue with real certainty without knowing which commands you're using and what your data looks like. But you can look up the time complexity of each method you're using and make sure it's reasonable.