kafka: Commit offsets failed with retriable exception. You should retry committing offsets

Question

[o.a.k.c.c.i.ConsumerCoordinator] [Auto offset commit failed for group consumer-group: Commit offsets failed with retriable exception. You should retry committing offsets.] []

Why does this error come in kafka consumer? what does this mean?

The consumer properties I am using are:

fetch.min.bytes:1
enable.auto.commit:true
auto.offset.reset:latest
auto.commit.interval.ms:5000
request.timeout.ms:300000
session.timeout.ms:20000
max.poll.interval.ms:600000
max.poll.records:500
max.partition.fetch.bytes:10485760

What is the reason for that error to come? I am guessing the consumer is doing duplicated work right now (polling same message again) because of this error.

I am neither using consumer.commitAsync() or consumer.commitSync()

Answer 1

Consumer gives this error in case if it catches an instance of RetriableException.

The reasons for it might be various:

• if coordinator is still loading the group metadata
• if the group metadata topic has not been created yet
• if network or disk corruption happens, or miscellaneous disk-related or network-related IOException occurred when handling a request
• if server disconnected before a request could be completed
• if the client's metadata is out of date
• if there is no currently available leader for the given partition
• if no brokers were available to complete a request

As you can see from the list above, all these errors could be temporary issues, that is why it is suggested to retry the request.