How can I retry failure messages from kafka?

Question

My spring-boot application(consumer) process messages from Apache Kafka. Periodically, massage can't process and consumer throw exception. Consumer commits offset anyway. Can I distinguish success messages from failure messages in Kafka? I think, I can't. Is it true? If it is true, I have the main question:

How can I retry failure messages? I know some ways but I'm not sure of their correctness.

1) Change offset to early. But in this way success messages will retry too.

2) When I catch an exception, I send this message to another topic(error-topic for example). But it looks difficult.

3) Something else(your variant)

Answer 1

If you want at-least once guarantee, a general pattern is as follows:

• Disable auto commit (set enable.auto.commit to false)
• Consume messages

• For each message:

  • If no errors, then commit offset
  • If error, retry as many times you wish
  • If successful, commit
  • If you want to give up, log or publish to an error queue (for analysis or later retry)

• Repeat