Menu
I am new to Kafka and I will appreciate very much clarification on the next case.
Kafka documentation says in the paragraph "Consumer Position":
"Our topic is divided into a set of totally ordered partitions, each of which is consumed by one consumer at any given time."
Based on statement above if few Consumer Groups subscribed to a topic and Producer will publish message to particular partition within this topic then only one Consumer can pull the message.
The question is how broadcast to many Consumer Groups could happen if only one Consumer can pull particular message?
241
You can't have multiple consumers that belong to the same group in one thread and you can't have multiple threads safely use the same consumer. One consumer per thread is the rule. To run multiple consumers in the same group in one application, you will need to run each in its own thread.
While Kafka allows only one consumer per topic partition, there may be multiple consumer groups reading from the same partition.
The same message can be consumed several times, either by different consumers, or indeed by the same consumer multiple times. Kafka uses the concepts of topics and partitions, where partitions are the main mechanism used to achieve its impressive scalability characteristics.
The consumers in a group divide the topic partitions as fairly amongst themselves as possible by establishing that each partition is only consumed by a single consumer from the group. When the number of consumers is lower than partitions, same consumers are going to read messages from more than one partition.
Only one consumer in a consumer group can pull the message. But all consumer groups get the messages.
So if you want all your consumers to get the messages, assign them different consumer groups. Each message goes to every consumer group, but within a group, it goes to only one consumer.
Read the Consumer section here.
98
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
