Read Kafka Topic from beginning

Question

I have a spring boot application, that keeps reading all messages from a specific topic. My problem is that I have an issue that when the application restarts, it needs to read all the messages from the topic again. I thought these two options combined would do it, but it is not working:

• resetOffsets: true
• startOffset: earliest

Here is my method;

 @StreamListener(myTopic)
 public void handle(@Payload Input input) {
    /** Do other stuff not related **/
 }

Here is my application.yaml

spring:
  cloud.stream:
    bindings:
      myTopic:
        destination: TOPIC
        content-type: application/**avro
        group: group-topic
        concurrency: 1
        resetOffsets: true
        startOffset: earliest

Answer 1

You need to change the group within your application.yaml to a new and unique group name (see below example where I set the consumer group to new-consumer-group-id:

spring:
  cloud.stream:
    bindings:
      myTopic:
        destination: TOPIC
        content-type: application/**avro
        group: new-consumer-group-id
        concurrency: 1
        resetOffsets: true
        startOffset: earliest

If you keep on using the same ConsumerGroup, the configurations resetOffset and startingOffset will not have any impact.

As an alternative, you could reset the offsets for each consumer group using the command line tool kafka-consumer-groups.sh. A template is shown below:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
 --execute --reset-offsets \
 --group groupA\
 --topic topicC \
 --partition 0 \
 --to-offset <insert number>