Redis expire on large set of keys

Question

My problem is: i have a set of values that each of them has to have an expire value. code:

set a:11111:22222 someValue
expire a:11111:22222 604800 \\usually equal a week

In a perfect world i would have put all those values in a hash and give each of them it's appropriate expire value, but redis does not allow expire on a hash fields.

problem is that i also have a process that need to get all those keys about once an hour

keys a:*

this command is really expensive and according to redis documentation can cause performance issues. I have about 25000-30000 keys at each given moment.

Does someone knows how can i solve such a problem? thumbs up it guarantee (-;
Roy

Answer 1

Let me propose an alternative solution.

Rather than asking Redis to scan all the keys, why not perform a background dump, and parse the dump to extract the keys? This way, there is zero impact on the Redis instance itself.

Parsing the dump file is not as scary as it sounds, because you can use the excellent redis-rdb-tools package:

https://github.com/sripathikrishnan/redis-rdb-tools

You can either convert the dump file into a json file, and then parse the json file, or use the Python API to extract the keys by yourself.