Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can multiple Kafka consumers read same message from the partition

We are planning to write a Kafka consumer(java) which reads Kafka queue to perform an action which is in the message.

As the consumers run independently, will the message is processed by only one consumer at a time? Else all the consumers process the same message as they have their own offset in the partition.

Please help me understand.

like image 615
shiv455 Avatar asked Feb 22 '16 18:02

shiv455


People also ask

What happens if there are more consumers than partitions in Kafka?

More consumers in a group than partitions means idle consumers. The main way we scale data consumption from a Kafka topic is by adding more consumers to a consumer group. It is common for Kafka consumers to do high-latency operations such as write to a database or a time-consuming computation on the data.

Are messages duplicated across partitions in Kafka?

Kafka may distribute messages by a key associated with each message, if a key is the same for some messages, all of them will be put in the same partition. Message brokers may offer some strategies to deliver messages: “only once” or “at least once”.

What happens if you have more consumers than partitions?

If there're more consumers in a group than paritions, some consumers will get no data. If you add new consumer instances to the group, they will take over some partitons from old members. If you remove a consumer from the group (or the consumer dies), its partition will be reassigned to other member.

How does Kafka consumer read from partition?

Reading records from partitions. Unlike the other pub/sub implementations, Kafka doesn't push messages to consumers. Instead, consumers have to pull messages off Kafka topic partitions. A consumer connects to a partition in a broker, reads the messages in the order in which they were written.


1 Answers

It depends on Group ID. Suppose you have a topic with 12 partitions. If you have 2 Kafka consumers with the same Group Id, they will both read 6 partitions, meaning they will read different set of partitions = different set of messages. If you have 4 Kafka cosnumers with the same Group Id, each of them will all read three different partitions etc.

But when you set different Group Id, the situation changes. If you have two Kafka consumers with different Group Id they will read all 12 partitions without any interference between each other. Meaning both consumers will read the exact same set of messages independently. If you have four Kafka consumers with different Group Id they will all read all partitions etc.

like image 172
Lukáš Havrlant Avatar answered Oct 21 '22 12:10

Lukáš Havrlant