Kafka Consumer default Group Id

Question

I'm working with Apache Kafka and its Java client and I see that messages are load balanced across different Kafka Consumers belonging to the same group (i.e. sharing the same group id).

In my application I need all consumers to read all messages.

So I have several questions:

• if I don't set any group id in the Consumer Properties, what group id will the Kafka Consumer be given?

• Is there a single default value?

• Does the client create a random value each time?

• Do I need to create a different id for each consumer to be sure that each one receives all messages?

EDIT: Thank you for your answers.

You are correct: if one doesn't set the consumer group id, Kafka should complain.

However, I have found out that if the group id is null, the Java client sets it to the empty string "" to avoid problems. So apparently that is the default value I was looking for.

Surprising all my consumers, even if I don't set their groupIds (and so they are all with groupId == "") seem to receive all the messages the producer writes.

I still can't explain this: any suggestions?

Answer 1

if I don't set any group id in the Consumer Properties, what group id will the Kafka Consumer be given?

The kafka consumer will not have any consumer group. Instead you will get this error : The configured groupId is invalid

Is there a single default value?

Yes, you can see the consumer.properties file of kafka for reference. The default consumer group id is: group.id=test-consumer-group

Does the client create a random value each time?

No, groupId seems to be mandatory for Java client starting Kafka 0.9.0.x consumers. You can refer to this JIRA: https://issues.apache.org/jira/browse/KAFKA-2648

Do I need to create a different id for each consumer to be sure that each one receives all messages?

Yes, if all consumers use the same group id, messages in a topic are distributed among those consumers. In other words, each consumer will get a non-overlapping subset of the messages. Having more consumers in the same group increases the degree of parallelism and the overall throughput of consumption. On the other hand, if each consumer is in its own group, each consumer will get a full copy of all messages.