Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to atomically delete millions of keys matching a pattern using pure Redis?

Tags:

redis

lua

Say I have millions of prefix:<numeric_id> keys.

I want to purge them all atomically.

How to atomically delete keys matching a pattern using Redis shows many options. Some use redis-cli or Bash script but I need to do it using my client, programmatically.

A Lua script approach is promising, but solutions with KEYS command fail with "too many elements to unpack" error.

How to achieve this?

like image 236
LeoMurillo Avatar asked Apr 22 '20 13:04

LeoMurillo


People also ask

How do I delete all keys matching a pattern in Redis?

Redis does not offer a way to bulk delete keys. You can however use redis-cli and a little bit of command line magic to bulk delete keys without blocking redis. This command will delete all keys matching users:* If you are in redis 4.0 or above, you can use the unlink command instead to delete keys in the background.

How do I flush all Redis?

To clear data of a DCS Redis 4.0 or 5.0 instance, you can run the FLUSHDB or FLUSHALL command in redis-cli, use the data clearing function on the DCS console, or run the FLUSHDB command on Web CLI. To clear data of a Redis Cluster instance, run the FLUSHDB or FLUSHALL command on every shard of the instance.

How can I get total number of keys in Redis?

The first command you can use to get the total number of keys in a Redis database is the DBSIZE command. This simple command should return the total number of keys in a selected database as an integer value. The above example command shows that there are 203 keys in the database at index 10.

Can Redis have duplicate keys?

You can use the DUMP and RESTORE commands to duplicate the key: use the DUMP command to serialize the value of a key. use the RESTORE command to restore the serialized value to another key.


2 Answers

The following Lua script uses SCAN command, so it deletes in chunks within the script - avoiding the "too many elements to unpack" error.

local cursor = 0
local calls = 0
local dels = 0
repeat
    local result = redis.call('SCAN', cursor, 'MATCH', ARGV[1])
    calls = calls + 1
    for _,key in ipairs(result[2]) do
        redis.call('DEL', key)
        dels = dels + 1
    end
    cursor = tonumber(result[1])
until cursor == 0
return "Calls " .. calls .. " Dels " .. dels

It returns how many times SCAN was called and how many keys were deleted.

Use as:

EVAL "local cursor = 0 local calls = 0 local dels = 0 repeat    local result = redis.call('SCAN', cursor, 'MATCH', ARGV[1])     calls = calls + 1   for _,key in ipairs(result[2]) do       redis.call('DEL', key)      dels = dels + 1     end     cursor = tonumber(result[1]) until cursor == 0 return 'Calls ' .. calls .. ' Dels ' .. dels" 0 prefix:1

Note it will block the server while running, so it is not advised for production as is.

For production, consider changing DEL for UNLINK. You can also return the cursor (instead of repeating inside the script until it is zero) and add COUNT parameter to SCAN to throttle (see Is there any recommended value of COUNT for SCAN / HSCAN command in REDIS?). This way you do it in chunks instead of one go, similar to How can I get all of the sets in redis?

Or you can do something more sophisticated using the approach stated in this answer: Redis `SCAN`: how to maintain a balance between newcomming keys that might match and ensure eventual result in a reasonable time?

like image 109
LeoMurillo Avatar answered Sep 20 '22 00:09

LeoMurillo


Lua is a great option as long as you are not using Redis Cluster or all the keys you want to delete are on the same shard.

If you need to delete keys from multi-shards you can still use Lua but you'll have to send the Eval command to all the shards "manually".

An alternative that does it for you, is using RedisGears (a Redis module), which allow you to write a cross cluster del command based on some criteria.

See example: https://oss.redislabs.com/redisgears/examples.html#delete-by-key-prefix

like image 32
Guy Korland Avatar answered Sep 21 '22 00:09

Guy Korland