Menu
How do I add delayed jobs on kafka? As I understand it doesn't deal with per message but per topic. My jobs have varying schedule in which I would like them to be consumed. Say one will be in the next 4 hours, another would be i Dec. 1, etc.
Does kafka have native support for this or other 3rd party ways to achieve the same?
I'm thinking of using Redis for the delayed queue instead, and push the job to kafka once its schedule has arrived but if possible I'd like to use only one dependency.
536
The usage of Apache Kafka is growing tremendously because of its unique design and high performance, but it lacks the support for delay queues and dead letter queues.
We can use Kafka as a Message Queue or a Messaging System but as a distributed streaming platform Kafka has several other usages for stream processing or storing data.
There is also another way to run retry logic with new versions of Spring Kafka. factory. setErrorHandler(new SeekToCurrentErrorHandler(new DeadLetterPublishingRecoverer(kafkaTemplate), 3)); With the config above, kafka consumer retries 3 times in case of error and publishes the message to “topicName + .
There is no notion of jobs in Kafka. It is just a dumb high performance message queueing service. Depending on your requirements you may consider storing the jobs in a storage that supports indexing by job execution time like some RDBMS. Then in some process periodically extract the jobs with execution times in some small range [last_check_time, current_time+lookahead_interval] and put them into a Kafka topic for eventual processing.
151
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
