Menu
I have a use case where i consume certain logs from a queue and hit some third party API's with some info from that log , in case the third party system is not responding properly i wish to implement a retry logic for that particular log .
I can add a time field and repush the message to the same queue and this message will get again consumed if its time field is valid i.e less than current time and if not then get pushed again into queue.
But this logic will add same log again and again until retry time is correct and the queue will grow unnecessarily.
Is there is better way to implement retry logic in Kafka ?
235
Power of retry mechanism actually lies in allowing the processing of messages multiple times. To implement this we need to add retryCount field in our class which will store the number of times the message needs to get processed.
Topic retry contains messages that have no value after processing, so in this case it is enough to configure retention, i.e. message lifetime. You just have to remember that retention should not be shorter than the longest possible processing time of a single message.
ERROR HANDLING IN CONSUMER In that case, consumer should retry to consume data within some time intervals. Kafka records are stored in the topics. If the topic is not found, consumer gets “Unknown topic or partition” error. The use case is very similar with the previous item.
The 'stateful' part means that the same consumer instance that is polling this topic partition knows how many retries it has done for this message, and hence when the retries are exceeded it can dead letter the message.
You can create several retry topics and push failed task there. For instance you can create 3 topics with different delays in mins and rotate the single failed task till the max attempt limit reached.
‘retry_5m_topic’ — for retry in 5 minutes
‘retry_30m_topic’ — for retry in 30 minutes
‘retry_1h_topic’ — for retry in 1 hour
See more for details: https://blog.pragmatists.com/retrying-consumer-architecture-in-the-apache-kafka-939ac4cb851a
157
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
