Spring and Kafka - Multiple Consumers with same Group ID - Only one consumer gets the messages

Question

I'm getting started with Apache Kafka using Spring Boot. And I want to achieve the fact that two instances of the same service consumes messages of the same topic in the round robin style.

So instance 1 receive the first message of the topic, instance 2 gets the second, instance 1 gets the third and on.

That's my current configuration:

spring:
  kafka:
    consumer:
      bootstrap-servers: rommel.local:9092
      group-id: core
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      enable-auto-commit: false
    producer:
      bootstrap-servers: rommel.local:9092
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
    listener:
      missing-topics-fatal: false
      ack-mode: record

This is my listener:

@KafkaListener(topics = "new-topic", groupId = "core")
    public void consume(String payload) {
        System.out.println("Data received: " + payload);
    }

The problem is, when I run two instances of my application, only one receives the messages of the topic.

I realized this in the logs:

Instance 1:
core: partitions assigned: [new-topic-0]

Instance 2:
core: partitions assigned: []

When I kill instance 1, then. the new-topic-0 is assigned to instance 2.

So how can I achieve the scenario of both instances getting messages of the same topic in round robin?

Thanks!!

Answer 1

First, let's get a few things straight:

1. In Kafka, the basic unit of parallelism is partition. If your topic has only one partition, no matter how many consumer instances / threads you configure, only one of them will recieve the message - the other instances will stay idle. If the instance currently processing messages, goes down, the partition will be assigned to other consumer instances / threads by a mechanism called "rebalancing" and that instance will then start processing the message. This is the behaviour you are observing here.

2. Strictly speaking, even with multiple partitions, it won't be round robin. The partitions will be equally distributed across the consumer threads / processes. So, each consumer instance / thread will have one or more partitions exclusively assigned to it and it will process the messages in those partitions only.

3. When you produce messages to the topic, where your messages will land up depends on the partition key you would use. That means the messages with the same partition key will always end up in the same partition (unless you increase the number of partitions after some time). This also means that the order of the messages will be maintained within a given partition and not across the partitions.

4. In older versions of Kafka, if you don't have a partition key when producing messages, the messages might get produced in round robin fashion across partitions. However, in recent versions of Kafka, even in the absence of a partition key, the messages will be batched together according to the producer configuration of the parameters batch.size and linger.ms before being sent out.