Horizontal scaling of consumers when the publisher provides sequenced messages

Question

In a distributed service oriented architecture, lets say I have a producer that send messages to a consumer using RMQ.

We decided then to horizontally scale the consuming part of our architecture by adding more consumers and we faced some limitations.

The publisher provide a sequence number in every message it sent. And it’s very important that the consumers process the messages based on the sequence number it has.

Every time that deal with a given resource, lets say A, the publisher will send RMQ messages that says "Hey lets do sequence 1 for A" and then "Hey lets do sequence 2 for A" and so on.

If for example the publisher provides 3 messages for A with sequences 1, 2 and 3 and the 3 messages are distributed to 3 different instances of our consumer. The message of sequence 2 is requeued until sequence 1 is well processed, same for sequence 3.

At the end the messages are all well processed, but after many retries! This causes some latency in our system as we retries many times if we’ve 100 sequences to consume.

A possible solution would be to make sure each set of sequences for a given resource has to be processed by the same consumer. But how can we achieve that?

How can I avoid the requeuing in order to make sure every instance of our consumer always get the messages for a given resource well ordered?

Answer 1

Try the following:

• Create a topic exchange
• Make your producer to send messages to it with topic per resource: topic A for resource A, topic B for resource B and so on.
• Make 1 consumer to listen to 1 topic. Use durable queue for each consumer, so the queue will survive consumer restarts. If you have producer -> one exchange -> 1 queue -> 1 consumer message path, the order of messages is guaranteed, see RabbitMQ broker semantics for details.
• Make your consumer to get 1 message at a time, and ack it only when processing is done.

This will hopefully give you sequential processing that you need and you don't need to re-queue messages.

Note that there is a possibility of receiving the same message twice - if your consumer died in between finishing processing task and acking the message.