How does Redis work when RAM starts filling up?

Question

I could be totally off, but my understanding of how cache stores used to work before they began to add persistence features, is that items would get expired based on their TTL. And if the store started to fill up available RAM, they would each have their algorithms for expiring the least "important" keys in the store.

Now I read that Redis has persistence features. But you can turn them off. Assuming you turn off persistence, what happen when RAM fills up? How does Redis decide what to expire?

I expect to have lots of data without TTLs and want to make sure it's safe to let Redis figure out what to expire.

Answer 1

I don't think the question is related to virtual memory management, but more about the expiration of the items in Redis, which is a totally different topic.

Contrary to memcached, Redis is not only a cache. So the user is supposed to choose about the item eviction policy using various mechanisms. You can evict all your items, or only a part of them.

The general policy should be selected in the configuration file with the maxmemory and maxmemory-policy parameters, decribed below:

# Don't use more memory than the specified amount of bytes. # When the memory limit is reached Redis will try to remove keys with an # EXPIRE set. It will try to start freeing keys that are going to expire # in little time and preserve keys with a longer time to live. # Redis will also try to remove objects from free lists if possible. # # If all this fails, Redis will start to reply with errors to commands # that will use more memory, like SET, LPUSH, and so on, and will continue # to reply to most read-only commands like GET. # # WARNING: maxmemory can be a good idea mainly if you want to use Redis as a # 'state' server or cache, not as a real DB. When Redis is used as a real # database the memory usage will grow over the weeks, it will be obvious if # it is going to use too much memory in the long run, and you'll have the time # to upgrade. With maxmemory after the limit is reached you'll start to get # errors for write operations, and this may even lead to DB inconsistency. # maxmemory <bytes>  # MAXMEMORY POLICY: how Redis will select what to remove when maxmemory # is reached? You can select among five behavior: # # volatile-lru -> remove the key with an expire set using an LRU algorithm # allkeys-lru -> remove any key accordingly to the LRU algorithm # volatile-random -> remove a random key with an expire set # allkeys->random -> remove a random key, any key # volatile-ttl -> remove the key with the nearest expire time (minor TTL) # noeviction -> don't expire at all, just return an error on write operations # # Note: with all the kind of policies, Redis will return an error on write #       operations, when there are not suitable keys for eviction. # #       At the date of writing this commands are: set setnx setex append #       incr decr rpush lpush rpushx lpushx linsert lset rpoplpush sadd #       sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby #       zunionstore zinterstore hset hsetnx hmset hincrby incrby decrby #       getset mset msetnx exec sort # # The default is: # maxmemory-policy volatile-lru  # LRU and minimal TTL algorithms are not precise algorithms but approximated # algorithms (in order to save memory), so you can select as well the sample # size to check. For instance for default Redis will check three keys and # pick the one that was used less recently, you can change the sample size # using the following configuration directive. # maxmemory-samples 3 

Then individual item expiration can be set using the following commands: EXPIRE EXPIREAT The per item expiration property is useful with volatile-* policies. Expiration can also be removed using PERSIST.

The expiration property adds a slight memory overhead, so it should be used only if required.

Finally, it is worth mentioning that a part of an object cannot be expired, only the whole object itself. For instance, a whole list or set corresponding to a key can be expired, but individual list or set items cannot.

Answer 2

Didier is right in indicating how this is to be done. Just indicating some additional elements (one of which seems to be omitted from his post):

1. Specify the max memory size to take up most of the available memory on that node (leave some for the OS and other processes and some buffer). This will make sure that the content is never paged and as such the operations are FAST.

2. If you are not setting TTLs / expiring keys through the application, then it is important that you use the "allkeys-lru" approach. Otherwise, Redis will not expire anything (because none of the keys are volatile), and you'll start getting errors once all the memory get's used up - basically you'll not be able to do any more set operations.

3. When using the LRU for removing keys, it is important to set the following setting:

   maxmemory-samples 10 

This is the number of samples that Redis will take and then remove the LRU key from amongst these. The default value is 3 but for all practical purposes this is too low - and may mean that older keys may still have survived. Setting this too high will be an overhead for Redis. You may want to experiment with this setting a bit before setting it. We are using a value of 10.