How Kafka consumer works if consumers are more that partitions

Question

Could anyone please explain and direct me to a link or resource to read about how the Kafka consumers work in below scenarios.

1. 1 consumer-group with 5 consumers and 1 topic with 3 partitions (how Kafka decides)

2. 1 Consumer-group with 5 consumers and 1 topic with 10 partitions (how would Kafka share load)

3. 2 consumer-groups with 1 consumer each and Kafka cluster of 2 servers where one topic is partitioned between node 1 and node 2, how duplications can be avoided when consumers from different groups subscribed to one partition.

The above scenarios may not be the best practice when configuring Kafka, but I'd like to understand how they need to be handled.

Answer 1

It's not Kafka itself to assign partitions, but one of the consumers. The first one joining a consumer group will be elected as sort of "leader" and we'll start assigning partitions to the other consumers. Of course, every time a new consumer joins the group, the Kafka "controller" let the leader consumer to know about that and it starts the rebalancing re-assigning partitions. It's the same when a consumer leaves a consumer group.

To confirm that the consumer is involved on that, the strategy for partition assignment is specified by the partition.assignment.strategy property in a consumer configuration. The default value is RangeAssignor while the other ones are RoundRobinAssignor and StickyAssignor. You can find more about how they work here:

https://kafka.apache.org/21/javadoc/org/apache/kafka/clients/consumer/RangeAssignor.html https://kafka.apache.org/21/javadoc/org/apache/kafka/clients/consumer/RoundRobinAssignor.html https://kafka.apache.org/21/javadoc/org/apache/kafka/clients/consumer/StickyAssignor.html

Said that, what happens specifically in your scenarios?

1. 3 consumers will get one partition each. The other 2 will be idle.
2. each consumer will get 2 partitions
3. Using different consumer groups mean pure pub/sub where the consumer groups get same messages. In your case with 1 topic and 2 partitions (on 2 brokers), the two consumers each in one different consumer group, will get the same messages from all 2 partitions. If consumers are part of different consumer groups you cannot avoid duplication, it's how Kafka works.