Kafka what will happen with message if a consumer group member goes down?

Question

Suppose we have a topic named Topic and a consumer group CG with three consumers. The group offset equals to 0.

The consumers start to read messages.

Read sequence:

1. Consumer 1 reads message 1.
2. Consumer 2 reads message 2.
3. Consumer 3 reads message 3.
4. Consumer 2 commits message 2.
5. Consumer 3 commits message 3.
6. Consumer 1 fails on processing of message 1 and goes down.

The question is: what will happen with message 1?

Or maybe I understand wrongly how consumers do read the messages as I am new to Kafka.

Version: Apache Kafka 2.4.0

Answer 1

You missed about partitions, Multiple consumers of same group will never consume messages from same partition

Suppose if you have the topic with three partitions(P0,P1,P2),and if you have three consumers(C1,C2,C3) of same group then each one will start consuming from each partition

If any consumer fails to submit offset and go down, then it will again start consuming the messages from previous offset (in your case 0)

Suppose if you have topic of 5 partition (P0,P1,P2,P3,P4) and of three consumers (C0,C1,C3) of same group. Then consumers will try to load balance equally by each taking two partition

C1 consumes from P0 and P1, and C2 consumes from P2 and P3 and remaining C3 consumes from P4.

Answer 2

Each consumer will have partitions assigned to them. Let's say we have 6 partitions:

Consumer 1: Partiton 1 & 2
Consumer 2: Partition 3 & 4
Consumer 3: Partition 5 & 6

When Consumer 1 goes down, the consumer group will re-balance and assign the free partitions to the other consumers, thus giving us the following setup:

Consumer 2: Partition 1 & 3 & 4
Consumer 3: Partition 2 & 5 & 6

The other consumers will start off from the last committed offset on that partition.