Recoverable PUB/SUB with redis

Question

I am building a PUB/SUB system backed by Redis.

I have one publisher and tons of subscribers. The subscribers are not that reliable, they can lose connection at any time and need to be able to "recover" from a lost connection.

There is a twist though, I would like my backlog capped at some number, meaning that a faulty subscriber should be able to recover only up to N messages.

The trivial solution is:

1. Publisher publishes message X
2. X is pushed onto a list RPUSH list message
3. Message is encoded to include its index in the list
4. Message is published to consumers (with the index embedded) PUBLISH channel encoded

If a consumer needs to re-establish:

1. It asks the redis for all the values in the list after the index it has and executes a PSUBSCRIBE atomically

---

Up to here we are all good.

My big question is, what if I want the backlog list to be capped at N items?

Is there any way I can keep an ever increasing index AND a capped backlog in the list?

Answer 1

How about this? To publish a message, do

LPUSH list message
LTRIM list 0 N
INCR global_index
PUBLISH channel global_index

When receiving a message over pub/sub and when starting up, clients will need to compare their latest index (this could also be kept in redis, or somewhere else) with global_index and read min(global_index - my_index, N) messages from the list to catch up (basically LRANGE list 0 (global_index - my_index)).