Kafka Commit Gaps

Question

I am trying to implement a loop where we will process the a batch of messages in loop, post successful processing we will commit the offset of the message, however for below scenario I am not able to find any tangible input.

Read: 1,2,3,4,5....10 I will process them in loop sequentially, if the message processing succeed then only the offset should be committed.

so imagine 1,2 succeeded but 3 failed, 4,5 success, 7,8 fail... if I only commit the offsets 1,2,4,5 what will happen to uncommitted offsets? 3,7,8...

Any idea will be read back again?

Answer 1

If your largest committed offset is 5 and you stop your consumer (or it crashes), the next time your consumer runs will start reading from offset 6 onward. Kafka does not provide a mechanism to deal with uncommitted individual offsets between committed offsets. Put differently, Kafka has no knowledge about any commit gaps. When the offset 5 has been committed, Kafka interprets this that your consumer has consumed all messages until offset 5.

To solve the problem you are describing, your application should be able to catch the failed messages and send them somewhere else (e.g. another Kafka topic). That way you can analyze the problematic messages and in case re-ingest them into your application.