How to improve Redis server's CPU usage?

Question

My goal is for our Redis server to hit about 80% CPU utilization in production. This will benefit our backend server design by ensuring we are not under-utilizing CPU while also leaving some headroom for growth and spikes.

While using Redis' own benchmark tool redis-benchmark, it's very easy to reach about 100% CPU usage:

    $ redis-benchmark -h 192.168.1.6 -n 1000000 -c 50

On this benchmark, we assigned 50 clients to push 1,000,000 requests to our Redis server.

But while using some other client tools(such as redis-lua or webdis), the maximum CPU usage is less than 60%.

I browsed some code in webdis and redis-lua. webdis depends on hiredis, and redis-lua is implemented in Lua, and depends on socket(lua-socket).

Are these clients too slow compared to the Redis benchmark and are unable to maximize the Redis CPU consumption?

I also browsed some code in redis-benchmark.c. The benchmark's main work is done in aeMain. It seems like that the redis-benchmark uses fast code from Redis, and my test clients(webdis and redis-lua) do not.

Currently my client have two choices:

1. Use redis-lua
2. Use a tool like webdis

However, these two do not maximize Redis' CPU utilization (less than 60%). Are there any other choices?

Or, is it possible to fully utilize redis-server outside of the redis-benchmark tool itself?

Answer 1

I doubt maximizing CPU usage of Redis will benefit your backend design. The right question is rather whether Redis is efficient enough to sustain your throughput at a given latency. Redis is a single-threaded server: at 80% CPU consumption, the latency will likely be very bad.

I suggest you measure latency while redis-benchmark is working to see if it is acceptable for your needs before trying to increase Redis CPU consumption. The --latency option of redis-cli can be used for this:

• launch redis-server
• try redis-cli --latency, note the avg value, stop it
• in another window, start the benchmark, and make sure it runs for a while
• try redis-cli --latency, note the avg value, stop it
• stop the benchmark
• compare the two avg values

Now, if you really want to increase Redis CPU consumption, you need either an efficient client program (like redis-benchmark), able to handle multiple connections concurrently, either multiple instances of your client program.

Lua is a fast interpreted language, but it is still an interpreted language. It will be one or two orders of magnitude slower than C code. Redis is much faster at parsing/generating its protocol than lua-redis, so you will not be able to saturate Redis with a unique Lua client (except if you use O(n) Redis commands - see later).

webdis is implemented in C, with an efficient client library, but has to parse the http/json protocols which happen to be more verbose and complex than Redis protocol. It likely consumes more CPU than Redis itself for most operations. So again, you will not saturate Redis with a single webdis instance.

Here are some examples to saturate Redis with multiple Lua clients.

If it is not already done, I suggest you had a look at the Redis benchmark page first.

If you run your benchmark on the same box as Redis:

The key point is to dedicate a core to Redis, and run the client programs on the other cores. On Linux, you can use the taskset command for this.

# Start Redis on core 0
taskset -c 0 redis-server redis.conf

# Start Lua programs on the other cores
for x in `seq 1 10` ; do taskset -c 1,2,3 luajit example.lua & done

The Lua program should use pipelining to maximize throughput and reduce system activity.

local redis = require 'redis'
local client = redis.connect('127.0.0.1', 6379)
for i=1,1000000 do
    local replies = client:pipeline(function(p)
    for j=1,1000 do
            local key = 'counter:'..tostring(j)
            p:incrby(key,1)
        end
    end)
end

On my system, the Lua program takes more than 4 times the CPU of Redis, so you need more than 4 cores to saturate Redis with this method (a 6 cores box should be fine).

If you run your benchmark on a different box than Redis:

Except if you run on CPU starved virtual machines, the bottleneck will likely be the network in that case. I don't think you can saturate Redis with anything less than a 1 GbE link.

Be sure to pipeline your queries as far as you can (see the previous Lua program) to avoid the network latency bottleneck, and reduce the cost of network interrupts on the CPU (filling ethernet packets). Try to run Redis on a core which is not bound to the network card (and processes network interrupts). You can use tools like htop to check this last point.

Try to run your Lua clients on various other machines of the network if you can. Again you will need a good number of Lua clients to saturate Redis (6-10 should be fine).

In some cases, a unique Lua process is enough:

Now, it is possible to saturate Redis with a single Lua client if each query is expensive enough. Here is an example:

local redis = require 'redis'
local client = redis.connect('127.0.0.1', 6379)

for i=1,1000 do
    local replies = client:pipeline(function(p)
        for j=1,1000 do
            p:rpush("toto",i*1000+j)
        end
    end)
end

N = 500000
for i=1,100000 do
    local replies = client:pipeline(function(p)
        for j=1,10 do
            p:lrange("toto",N, N+10)
        end
    end)
end

This program populates a list with 1M items, and then uses lrange commands to fetch 10 items from the middle of the list (worst case for Redis). So each time a query is executed, 500K items are scanned by the server. Because only 10 items are returned, they are fast to parse by lua-redis which will not consume CPU. In this situation, all the CPU consumption will be on server side.

Final words

There are probably faster Redis clients than redis-lua:

• https://github.com/agladysh/lua-hiredis (based on hiredis)
• https://github.com/agladysh/ljffi-hiredis (based on hiredis, using luajit FFI)

You may want to try them.