Enforce Unique consumer group id in Kafka

Question

We are making a Kafka Queue into which messages are being published from source system. Now multiple consumers can connect to this queue to read messages.

While doing this the consumers have to specify a groupId based on which the messages are distributed , if two apps have the same groupId both of them won't get the messages.

Is there a way i can enforce every app. to have a unique consumer group Id?

Answer 1

Kafka is not a naming registry, but there are many other strategies that you might use to assign unique group names without a coordinator:

• Name the group based on the application. Two apps will each get all the data since their names are unique.
• Name the group based on the data store that they are writing to (e.g. jdbc connection string). If you start an app instance for a different database, it will have it's own offsets
• Name the group based on the (canonical) class name that implements the consumer. If your app has two consumers implemented in different Java classes, they'll get different groupIds
• Assign a random value (e.g. UUID) to the group. On ever restart a new group will be created.

Answer 2

Even though I used uuid suffix for group id for some time it looked weird and polluted consumer list in kafka tool. For me random group id was needed for topics that kind of needed everywhere e.g. currency rates. Let say I have k8s deployment with 3 pods and each one needed latest rates to serve data for players in different currencies. Then I was reading doc again

group.id

A unique string that identifies the consumer group this consumer belongs to. This property is required if the consumer uses either the group management functionality by using subscribe(topic) or the Kafka-based offset management strategy.

and realised that for this use case you don't need neither group management (because this topic is not really scaled if you need it everywhere) nor offset management (because with random group kafka it is always new group with no offsets stored). So I decided to use single partition topic (i.e. partition 0) and assign(0) explicitly instead of subscribe() in which case you can skip group.id property at all.