Redis is slow to get large strings

Question

I'm kind of a newb with Redis, so I apologize if this is a stupid question.

I'm using Django with Redis as a cache.

I'm pickling a collection of ~200 objects and storing it in Redis.

When I request the collection from Redis, Django Debug Toolbar is informing me that the request to Redis is taking ~3 seconds. I must be doing something horribly wrong.

• The server has 3.5GB of ram, and it looks like Redis is currently using only ~50mb, so I'm pretty sure it's not running out of memory.

• When I get the key using the redis-cli it takes just as long as when I do it from Django

• Running strlen on the key from redis-cli I'm informed that the length is ~20 million (Is this too large?)

What can I do to have Redis return the data faster? If this seems unusual, what might be some common pitfalls? I've seen this page on latency problems, but nothing has really jumped out at me yet.

I'm not sure if it's a really bad idea to store a large amount of data in one key, or if there's just something wrong with my configuration. Any help or suggestions or things to read would be greatly appreciated.

Answer 1

It's most likely just the size of the string. I'd look at whether your objects are being serialized efficiently.

Answer 2

Redis is not designed to store very large objects. You are not supposed to store your entire collection in a single string in Redis, but rather use Redis list or set as a container for your objects.

Furthermore, the pickle format is not optimized for space ... you would need a more compact format. Protocol Buffers, MessagePack, or even plain JSON, are probably better for this. You should consider to apply a light compression algorithm before storing your data (like Snappy, LZO, Quicklz, LZF, etc ...).

Finally, the performance is probably network bound. On my machine, retrieving a 20 MB object from Redis takes 85 ms (not 3 seconds). Now, if I run the same test using a remote server, it takes 1.781 seconds, which is expected on this 100 Mbit/s network. The duration is fully dependent on the network bandwidth.

Last point: be sure to use a recent Redis version - a number of optimization have been done to deal with large objects.