Menu
It seems Kafka only allows consumers to pull messages from the leader broker (the broker on which the leader partition running). The followers only sync with the leader, and consumers CANNOT read messages from the followers.
Why NOT let consumers also pull messages from followers?
I think if the consumer can read messages from followers, we can have load balancing for reading from all broker servers, and the performance of the whole system should be better. Say, the leader is too busy, and the bandwidth of the leader becomes a bottleneck. If we can pull messages from the followers, the performance should be better.
473
Consumers can now consume messages directly from follower replicas, and they no longer need to connect to the leader replica.
If the consumer crashes or is shut down, its partitions will be re-assigned to another member, which will begin consumption from the last committed offset of each partition. If the consumer crashes before any offset has been committed, then the consumer which takes over its partitions will use the reset policy.
Consumer offset is recorded in Kafka so if the consumer processing the partition in the consumer group goes down and when the consumer comes back, the consumer will read the offset to start reading the messages from the topic from where it is left off. This avoids duplication in message consumption.
Kafka assigns the partitions of a topic to the consumer in a group, so that each partition is consumed by exactly one consumer in the group. Kafka guarantees that a message is only ever read by a single consumer in the group. Consumers can see the message in the order they were stored in the log.
There is no leader broker in Kafka only leader partitions. A topic is made up of 1 or more partitions. A Kafka broker has multiple partitions, some of which are leaders and some of which are replicas from partitions on the other Kafka brokers in the cluster. This way each broker has some active partitions and so one busy partition on one broker does not make as big of an negative impact on the performance of the other partitions on the remaining brokers.
82
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
