Menu
I have a producer which writes messages to a topic/partition. To maintain ordering, i would like to go with single partition and I want 12 consumers to read all the messages from this single partition(no consumer group, all the messages should go to all consumers). Is this achievable? I read some forums that only one consumer can read per partition.
977
Kafka consumers are typically part of a consumer group . When multiple consumers are subscribed to a topic and belong to the same consumer group, each consumer in the group will receive messages from a different subset of the partitions in the topic.
No two consumers of the same group-id would be assigned to the same partition. Suppose, there is a topic with 4 partitions and two consumers, consumer-A and consumer-B wants to consume from it with group-id “app-db-updates-consumer”. As shown in the diagram, Kafka would assign: partition-1 and partition-2 to consumer-A.
If there're more paritions than consumers in a group, some consumers will consume data from more than one partition. If there're more consumers in a group than paritions, some consumers will get no data. If you add new consumer instances to the group, they will take over some partitons from old members.
In some cases, the producer will direct messages to specific partitions by using the message key and a partitioner that will generate a hash of the key and map it to a specific partition. Kafka is able to seamlessly handle multiple producers that are using many topics or the same topic.
You may use SimpleConsumer to achieve exactly what you are asking - no consumer groups, all consumers can read a single partition. However this approach means you have to handle offset storing and broker failure handling yourself.
Another option is to use high level consumer with different consumer groups (you could just assign a random UUID to each consumer). This way you'll be able to consume one topic/partition with all consumers and be able to commit offsets and handle broker outage.
The rule "only a single consumer can consume a topic/partition" applies only to consumer groups, e.g. only one consumer IN GROUP can consume one topic/partition simultaneously.
135
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
