Concurrent priority queue in redis?

Question

I would like to implement a concurrent priority queue in Redis, with multiple processes on different machines adding items (with scores) and multiple other processes popping these items, lowest score first.

A simple queue can be implemented with LPUSH and RPOP.

Using a ZSET, I can add the items using ZADD and pop them with ZRANGE and ZREM, as long as there is only one reader.

For multiple readers I think I need something like ZPOP which combines ZRANGE and ZREM in a single atomic operation. Otherwise two readers may get the same item from ZRANGE before either can ZREM it. Retrying if ZREM returns 0 would work but is not desirable.

Is there some way I can do this using the current Redis commands? Is there any reason this hasn't been added to Redis already? It seems like it would be a pretty simple command to implement.

Answer 1

You can guarantee atomicity if you use a Lua script that does the ZRANGE & ZREM or with a MULTI/EXEC block. This will prevent multiple workers from interfering with each other.

I assume that ZPOP wasn't put in in the first place because it isn't a common use case and, when needed, it can be easily scripted.

Answer 2

you can use redis command: watch

WATCH zset
element = ZRANGE zset 0 0
MULTI
ZREM zset element
EXEC

if exec fails (return a null reply), just repeat those commands.