Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What Happens when there is only one partition in Kafka topic and multiple consumers?

Tags:

apache-kafka

I have a Kafka Topic with only one partition and I am not getting what will happen in following cases? How messages will be delivered to consumers?

  1. If all consumers are in same group
  2. If all consumers are in different group

enter image description here I am not sure if consumers will receive unique messages or duplicate ones.

like image 364
hard coder Avatar asked Oct 22 '18 17:10

hard coder


People also ask

Can multiple consumers read from a single partition Kafka?

So the rule in Kafka is only one consumer in a consumer group can be assigned to consume messages from a partition in a topic and hence multiple Kafka consumers from a consumer group can not read the same message from a partition.

How many partitions can a Kafka topic have?

But here are a few general rules: maximum 4000 partitions per broker (in total; distributed over many topics) maximum 200,000 partitions per Kafka cluster (in total; distributed over many topics) resulting in a maximum of 50 brokers per Kafka cluster.

Can multiple producers write to same partition in Kafka?

In some cases, the producer will direct messages to specific partitions by using the message key and a partitioner that will generate a hash of the key and map it to a specific partition. Kafka is able to seamlessly handle multiple producers that are using many topics or the same topic.

What is the purpose of partition in Kafka?

Kafka Partitioning Partitioning takes the single topic log and breaks it into multiple logs, each of which can live on a separate node in the Kafka cluster. This way, the work of storing messages, writing new messages, and processing existing messages can be split among many nodes in the cluster.


1 Answers

Each Consumer subscribes to a/more partition in a topic. And each consumer belongs to a consumer group. Below are two scenarios:

  1. When all consumers belong to the same group : Each consumer will try to subscribe to a different partition. In case,if there is only one partition, only one consumer will get the messages, while other consumers will be idle.

  2. When all consumers belong to the different consumer group: Each consumer will get the messages from all partitions. Partition subscription is based on the consumer groups.

It depends on the consumer groups. Consumers within the same consumer group don't read the data again from the same partitions once the read offsets have been committed.

like image 118
Nishu Tayal Avatar answered Nov 10 '22 21:11

Nishu Tayal