Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kafka multiple consumers for a partition

I have a producer which writes messages to a topic/partition. To maintain ordering, i would like to go with single partition and I want 12 consumers to read all the messages from this single partition(no consumer group, all the messages should go to all consumers). Is this achievable? I read some forums that only one consumer can read per partition.

like image 977
skumar Avatar asked Jun 04 '15 15:06

skumar


People also ask

How does Kafka handle multiple consumers?

Kafka consumers are typically part of a consumer group . When multiple consumers are subscribed to a topic and belong to the same consumer group, each consumer in the group will receive messages from a different subset of the partitions in the topic.

How do you set multiple consumers in Kafka?

No two consumers of the same group-id would be assigned to the same partition. Suppose, there is a topic with 4 partitions and two consumers, consumer-A and consumer-B wants to consume from it with group-id “app-db-updates-consumer”. As shown in the diagram, Kafka would assign: partition-1 and partition-2 to consumer-A.

What happens if there are more partitions than consumers?

If there're more paritions than consumers in a group, some consumers will consume data from more than one partition. If there're more consumers in a group than paritions, some consumers will get no data. If you add new consumer instances to the group, they will take over some partitons from old members.

Can multiple producers write to same Kafka partition?

In some cases, the producer will direct messages to specific partitions by using the message key and a partitioner that will generate a hash of the key and map it to a specific partition. Kafka is able to seamlessly handle multiple producers that are using many topics or the same topic.


1 Answers

You may use SimpleConsumer to achieve exactly what you are asking - no consumer groups, all consumers can read a single partition. However this approach means you have to handle offset storing and broker failure handling yourself.

Another option is to use high level consumer with different consumer groups (you could just assign a random UUID to each consumer). This way you'll be able to consume one topic/partition with all consumers and be able to commit offsets and handle broker outage.

The rule "only a single consumer can consume a topic/partition" applies only to consumer groups, e.g. only one consumer IN GROUP can consume one topic/partition simultaneously.

like image 135
serejja Avatar answered Sep 28 '22 06:09

serejja