Publish/Subscribe reliable messaging: Redis VS RabbitMQ

Question

Background

I am making a publish/subscribe typical application where a publisher sends messages to a consumer.

The publisher and the consumer are on different machines and the connection between them can break occasionally.

Objective

The goal here is to make sure that no matter what happens to the connection, or to the machines themselves, a message sent by a publisher is always received by the consumer.

Ordering of messages is not a must.

Problem

According to my research, RabbitMQ is the right choice for this scenario:

• Redis Vs RabbitMQ as a data broker/messaging system in between Logstash and elasticsearch

However, although RabbitMQ has a tutorial about publish and subscriber this tutorial does not present us to persistent queues nor does it mention confirms which I believe are the key to making sure messages are delivered.

On the other hand, Redis is also capable of doing this:

• http://abhinavsingh.com/customizing-redis-pubsub-for-message-persistence-part-2/

but I couldn't find any official tutorials or examples and my current understatement leads to me to believe that persistent queues and message confirms must be done by us, as Redis is mainly an in memory-datastore instead of a message broker like RabbitMQ.

Questions

1. For this use case, which solution would be the easiest to implement? (Redis solution or RabbitMQ solution?)
2. Please provide a link to an example with what you think would be best!

Answer 1

Background

I originally wanted publish and subscribe with message and queue persistence.

This in theory, does not exactly fit publish and subscribe:

• this pattern doesn't care if the messages are received or not. The publisher simply fans out messages and if there are any subscribers listening, good, otherwise it doesn't care.

Indeed, looking at my needs I would need more of a Work Queue pattern, or even an RPC pattern.

Analysis

People say both should be easy, but that really is subjective.

RabbitMQ has a better official documentation overall with clear examples in most languages, while Redis information is mainly in third party blogs and in sparse github repos - which makes it considerably harder to find.

As for the examples, RabbitMQ has two examples that clearly answer my questions:

• Work queues
• RPC example

By mixing the two I was able to have a publisher send to several consumers reliable messages - even if one of them fails. Messages are not lost, nor forgotten.

Downfall of rabbitMQ:

• The greatest problem of this approach is that if a consumer/worker crashes, you need to define the logic yourself to make sure that tasks are not lost. This happens because once a task is completed, following the RPC pattern with durable queues from Work Queues, the server will keep sending messages to the worker until it comes back up again. But the worker doesn't know if it already read the reply from the server or not, so it will take several ACK from the server. To fix this, each worker message needs to have an ID, that you save to the disk (in case of failure) or the requests must be idempotent.
• Another issue is that if the connection is lost, the clients blow up with errors as they cannot connect. This is also something you must prepare in advance.

As for redis, it has a good example of durable queues in this blog:

• https://danielkokott.wordpress.com/2015/02/14/redis-reliable-queue-pattern/

Which follows the official recommendation. You can check the github repo for more info.

Downfall of redis:

• As with rabbitmq, you also need to handle worker crashes yourself, otherwise tasks in progress will be lost.
• You have to do polling. Each consumer needs to ask the producer if there are any news, every X seconds.

This is, in my opinion, a worst rabbitmq.

Conclusion

I ending up going with rabbitmq for the following reasons:

1. More robust official online documentation, with examples.
2. No need for consumers to poll the producer.
3. Error handling is just as simple as in redis.

With this in mind, for this specific case, I am confident in saying that redis is a worst rabbitmq in this scenario.

Hope it helps.