Mongo tailable cursors vs Redis pub/sub

Question

I'm backing a real-time websocket server application with MongoDB.

The client base is growing, and single-threaded performance is no longer enough. I need a pub/sub layer to distribute messages across threads.

I would normally go for Redis, but since the app already uses MongoDB, I could avoid the dependency using tailable cursors. However, I worry about performance.

How does MongoDB's tailable cursor performance compare to that of Redis for a pub/sub architecture?

Answer 1

Actually, they are very different beasts.

A MongoDB tailable cursor would work a bit like a queue. It can work with a capped collection so you do not have to explicitly delete items in the collection. It is quite efficient, but keep in mind that MongoDB will lock the whole collection (the database actually) at each write operation, so it limits the scalability. Another scalability limitation is the number of connections. Each client connection will add a connection thread in the mongod servers (or mongos).

Still you can expect tens of thousands of items per second without major problems, which may be enough for a range of applications.

On the other hand, Redis can generally handle much more connections simultaneously, because each connection does not create a thread (Redis is a single-theaded event loop). It is also extremely CPU efficient, because it does not queue at all the items. With Redis pub/sub, the items are propagated to the subscribers in the same event loop iteration than the publication. The items are not even stored in memory, Redis does not even have a single index to maintain. They are only retrieved from a socket buffer to be pushed in another socket buffer.

However, because there is no queuing, delivery of Redis pub/sub messages is not guaranteed at all. If a subscriber is down when a message is published, the message will be lost for this subscriber.

With Redis, you can expect hundreds of thousands of items per second on a single core, especially if you use pipelining, and multiple publication clients.