In a publish/subscribe model in microservices, how to receive/consume a message only once per service type

Question

We are designing for a microservices architecture model where service A publishes a message and services B, and C would like to receive/consume the message. However, for high availability multiple instances of services B and C are running at the same time. Now the question is how do we design such that only one service instance of B and one service instance of C receive the message and not all the other service instances.

As far as I know about RabbitMQ, it is not easy to achieve this behavior. I wonder if Kafka or any other messaging framework has a built-in support for this scenario, which I believe should be very common in a microservices architecture.

Answer 1

Kafka has a feature called Consumer Groups that does exactly what you describe.

Every identical instance of B can declare its group.id to be the same string (say "serviceB") and Kafka will ensure that each instance gets assigned a mutually exclusive set of topic partitions for all the topics it subscribes to.

Since all instances of C will have a different group.id (say "serviceC") then they will also get the same messages as the instances of B but they will be in an independent Consumer Group so messages go only to 1 of N instances of C, up to maximum number of instances which is the total number of topic partitions.

You can dynamically and independently scale up or down the number of instances of B and C. If any instance dies, the remaining instances will automatically rebalance their assigned topic partitions and take over processing of the messages for the instance that died.

Data never has to be stored more than once so there is still one single commit log or "source of truth" for all these service instances.

Answer 2

Kafka has built-in support for this scenario.

You can create two Consumer Groups, one for B, and the other for C. Both Consumer Groups subscribe messages from A.

Any message published by A will be sent to both groups. However, only one member of each group can receive the message.