How slow is Redis when full and evicting keys ? (LRU algorithm)

Question

I am using Redis in a Java application, where I am reading log files, storing/retrieving some info in Redis for each log. Keys are IP addresses in my log file, which mean that they are always news keys coming, even if the same appears regularly.

At some point, Redis reaches its maxmemory size (3gb in my case), and starts evicting some keys. I use the "allkeys-lru" settings as I want to keep the youngest keys.

The whole application then slows a lot, taking 5 times longer than at the beginning. So I have three questions:

• is it normal to have such a dramatic slowdown (5 times longer)? Did anybody experience such slowdown? If not, I may have another issue in my code (improbable as the slowdown appears exactly when Redis reaches its limit)
• can I improve my config ? I tried to change the maxmemory-samples setting without much success
• should I consider an alternative for my particular problem? Is there a in-memory DB that could handle evicting keys with better performances ? I may consider a pure Java object (HashMap...), even if it doesn't look like a good design.

edit 1: we use 2 DBs in Redis

edit 2: We use redis 2.2.12 (ubuntu 12.04 LTS). Further investigations explained the issue: we are using db0 and db1 in redis. db1 is used much less than db0, and keys are totally different. When Redis reaches max-memory (and LRU algo starts evicting keys), redis does remove almost all db1 keys, which slows drastically all calls. This is a strange behavior, probably unusual and maybe linked to our application. We fixed the issue by moving to another (better) memory mechanism for keys that were loaded in db1.

thanks !

Answer 1

I'm not convinced Redis is the best option for your use case.

Redis "LRU" is only a best effort algorithm (i.e. quite far from an exact LRU). Redis tracks memory allocations and knows when it has to free some memory. This is checked before the execution of each command. The mechanism to evict a key in "allkeys-lru" mode consists in choosing maxmemory-samples random keys, comparing their idle time, and select the most idle key. Redis repeats these operations until the used memory is below maxmemory.

The higher maxmemory-samples, the more CPU consumption, but the more accurate result.

Provided you do not explicitly use the EXPIRE command, there is no other overhead to be associated with key eviction.

Running a quick test with Redis benchmark on my machine results in a throughput of:

• 145 Kops/s when no eviction occurs
• 125 Kops/s when 50% eviction occurs (i.e. 1 key over 2 is evicted).

I cannot reproduce the 5 times factor you experienced.

The obvious recommendation to reduce the overhead of eviction is to decrease maxmemory-samples, but it also means a dramatic decrease of the accuracy.

My suggestion would be to give memcached a try. The LRU mechanism is different. It is still not exact (it applies only on a per slab basis), but it will likely give better results that Redis on this use case.