Why a 500MB Redis dump.rdb file takes about 5.0GB memory?

Question

Actually, I have 3 Redis instances and I put them together into this 500MB+ dump.rdb. The Redis server can read this dump.rdb and it seems that everything is ok. Then I notice that redis-server cost more than 5.0GB memory. I don't know why.

Is there anything wrong with my file? My db has about 3 million keys, values for each key is a list contains about 80 integers.

I use this METHOD to put 3 instance together.

PS:Another dump.rdb with the same size and same key-value structure cost only 1GB memory.

And my data looks like keyNum->{num1, num2, num3,......}. All numbers is between 1 and 4,000,000. So should I use List to store them? For now, I use lpush(k, v). Did this way cost too much?

Answer 1

The ratio of memory to dump size depends on the data types Redis uses internally.

For small objects (hashes, lists and sortedsets), redis uses ziplists to encode data. For small sets made of integers, redis uses Intsets. ZipLists and IntSets are stored on disk in the same format as they are stored in memory. So, you'd expect a 1:1 ratio if your data uses these encodings.

For larger objects, the in-memory representation is completely different from the on-disk representation. The on-disk format is compressed, doesn't have pointers, doesn't have to deal with memory fragmentation. So, if your objects are large, a 10:1 memory to disk ratio is normal and expected.

If you want to know which objects eat up memory, use redis-rdb-tools to profile your data (disclaimer: I am the author of this tool). From there, follow the memory optimization notes on redis.io, as well as the memory optimization wiki entry on redis-rdb-tools.