How to atomically delete millions of keys matching a pattern using pure Redis?

Question

Say I have millions of prefix:<numeric_id> keys.

I want to purge them all atomically.

How to atomically delete keys matching a pattern using Redis shows many options. Some use redis-cli or Bash script but I need to do it using my client, programmatically.

A Lua script approach is promising, but solutions with KEYS command fail with "too many elements to unpack" error.

How to achieve this?

Answer 1

The following Lua script uses SCAN command, so it deletes in chunks within the script - avoiding the "too many elements to unpack" error.

local cursor = 0
local calls = 0
local dels = 0
repeat
    local result = redis.call('SCAN', cursor, 'MATCH', ARGV[1])
    calls = calls + 1
    for _,key in ipairs(result[2]) do
        redis.call('DEL', key)
        dels = dels + 1
    end
    cursor = tonumber(result[1])
until cursor == 0
return "Calls " .. calls .. " Dels " .. dels

It returns how many times SCAN was called and how many keys were deleted.

Use as:

EVAL "local cursor = 0 local calls = 0 local dels = 0 repeat    local result = redis.call('SCAN', cursor, 'MATCH', ARGV[1])     calls = calls + 1   for _,key in ipairs(result[2]) do       redis.call('DEL', key)      dels = dels + 1     end     cursor = tonumber(result[1]) until cursor == 0 return 'Calls ' .. calls .. ' Dels ' .. dels" 0 prefix:1

Note it will block the server while running, so it is not advised for production as is.

For production, consider changing DEL for UNLINK. You can also return the cursor (instead of repeating inside the script until it is zero) and add COUNT parameter to SCAN to throttle (see Is there any recommended value of COUNT for SCAN / HSCAN command in REDIS?). This way you do it in chunks instead of one go, similar to How can I get all of the sets in redis?

Or you can do something more sophisticated using the approach stated in this answer: Redis `SCAN`: how to maintain a balance between newcomming keys that might match and ensure eventual result in a reasonable time?

Answer 2

Lua is a great option as long as you are not using Redis Cluster or all the keys you want to delete are on the same shard.

If you need to delete keys from multi-shards you can still use Lua but you'll have to send the Eval command to all the shards "manually".

An alternative that does it for you, is using RedisGears (a Redis module), which allow you to write a cross cluster del command based on some criteria.

See example: https://oss.redislabs.com/redisgears/examples.html#delete-by-key-prefix