Menu
I see references to Kafka Consumer Group Coordinators and Consumer Group Leaders...
What is the difference?
What is the benefit from separating group management into two different sets of responsibilities?
970
Group Coordinator The coordinator uses an internal Kafka topic to keep track of group metadata. In a typical Kafka cluster, there will be multiple group coordinators. This allows for multiple consumer groups to be managed efficiently.
The group leader is responsible for executing rebalance activity. The group leader will take a list of current members, assign partitions to them and send it back to the coordinator. The Coordinator then communicates back to the members about their new partitions.
A consumer group is a set of consumers which cooperate to consume data from some topics. The partitions of all the topics are divided among the consumers in the group.
As, there are only two topic-partitions available, but three consumers.
The consumer group coordinator is one of the brokers while the group leader is one of the consumer in a consumer group.
The group coordinator is nothing but one of the brokers which receives heartbeats (or polling for messages) from all consumers of a consumer group. Every consumer group has a group coordinator. If a consumer stops sending heartbeats, the coordinator will trigger a rebalance.
It gives you more flexible/extensible assignment policies without rebooting the broker.
The key point of this separation is that group leader is responsible for computing the assignments for the whole group.
It means that this assignment strategy can be configured on a consumer (see partition.assignment.strategy consumer config parameter).
If a partitions assignment was handled by a consumer group coordinator, it would be impossible to configure a custom assignment strategy without rebooting the broker.
For more details see Kafka Client-side Assignment Proposal.
From the "Kafka The Definitive Guide" [Narkhede, Shapira & Palino, 2017]:
When a consumer wants to join a consumer group, it sends a
JoinGrouprequest to the group coordinator. The first consumer to join the group becomes the group leader. The leader receives a list of all consumers in the group from the group coordinator (this will include all consumers that sent a heartbeat recently and are therefore considered alive) and it is responsible for assigning a subset of partitions to each consumer. It uses an implementation of thePartitionAssignorinterface to decide which partitions should be handled by which consumer.[...] After deciding on the partition assignment, the consumer leader sends the list of assignments to the
GroupCoordinatorwhich sends this information to all the consumers. Each consumer only sees his own assignment - the leader is the only client process that has the full list of consumers in the group and their assignments. This process repeats every time a rebalance happens.
133
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
