Menu
I am using Spring Kafka consumer which fetches messages from a topic and persist them into a db. If a failure condition is met , say for instance the db is unavailable , does kafka consumer library provide mechanism to retry ? If it does , is there a way to set different retry intervals such as 1st retry should be done after 5 mins , 2nd after 30 mins , 3rd after 1 hr etc.
417
Retries happen within the consumer poll for the batch. Consumer poll must complete before poll timeout, containing all retries, and total processing time (including REST calls & DB calls), retry delay and backoff, for all records in the batch. Default poll time is 5 minutes for 500 records in the batch.
The retries setting determines how many times the producer will attempt to send a message before marking it as failed. The default values are: 0 for Kafka <= 2.0.
Processing failed messages can be achieved by cloning the message and republishing it to one of retry topics with updated information about attempt number and next retry timestamp. Consumers of retry topics should block the thread unless it is time to process the message.
In order to disable retries in the STCEH, use a FixedBackOff with 0 retries. The default BackOff is 9 retries (10 attempts) with no back off.
Spring Kafka is shipped with the RetryingMessageListenerAdapter and RetryingAcknowledgingMessageListenerAdapter. If you use @KafkaListener, you can supply AbstractKafkaListenerContainerFactory with the RetryTemplate. And the last one can be injected with any custom RetryPolicy and BackOffPolicy from the Spring Retry project:
https://docs.spring.io/spring-kafka/reference/html/#retrying-deliveries
Also bear in mind that since version 2.0, there is transaction support in Spring Kafka, based on such one in the Apache Kafka 0.11.x.x:
https://docs.spring.io/spring-kafka/reference/html/#transactions
89
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
