why python using redis is so slow?

Question

I setup redis on my cloud server,and run redis-benchmark，the result is:

SET: 90909.09 requests per second GET: 91743.12 requests per second

but when I using python do the set/get in loop:

import datetime
import redis 
r = redis.Redis(host='127.0.0.1', port=6379, db=1)
mapData = {}
begin = datetime.datetime.now()
for i in range(8000):
        mapData[i] = r.get([str(i)])
end = datetime.datetime.now()
print end-begin

the result just 8000 requests per second. how can I improve it?

Answer 1

Start by reading http://redis.io/topics/benchmarks

In a nutshell, your python program generates too many roundtrips to Redis on a single connection, so you pay for the network latency at each iteration. My suggestions:

• use pipelining - examples in Python here
• use mget to retrieve several keys in one command
• do more than 8000 iterations, this is too low to assess the performance

Note that you will not reach the performance of redis-benchmark in Python, but you should be able to achieve more than 50K get operations per second with pipelining or mget.