Menu
If I have a RabbitMQ consumer that retrieves 100 messages in bulk, but then it crashes before it can mark those messages as processed, are those messages lost? I want every message in the queue to be processed at least once. What's the recommended approach to deal with consumers that crash before they've acknowledged messages?
Does RabbitMQ put them back on the queue somehow, or what do I need to do to make it happen?
846
When RabbitMQ quits or crashes it will forget the queues and messages unless you tell it not to. Two things are required to make sure that messages aren't lost: we need to mark both the queue and messages as durable.
Without acknowledgements, message loss is possible during publish and consume operations and only at most once delivery is guaranteed.
RabbitMQ has a plugin for consistent hash exchange. Using that exchange, and one consumer per queue, we can achieve message order with multiple consumers. The hash exchange distributes routing keys among queues, instead of messages among queues. This means all messages with the same routing key will go the same queue.
In standard queues, messages are retained for at least 72 hours and will be deleted 72 hours later.
What's the recommended approach to deal with consumers that crash before they've acknowledged messages?
Let the rabbitmq do everything for you - messages that are not acknowledged are re-queued and will be delivered again to another (or even same) consumer.
Does RabbitMQ put them back on the queue somehow, or what do I need to do to make it happen?
See answer to first question. Simply don't acknowledge the messages before they're processed. This means also making sure that auto_ack flag is not set!
If I have a RabbitMQ consumer that retrieves 100 messages in bulk, but then it crashes before it can mark those messages as processed, are those messages lost?
See answer above - they are lost if they're automatically acknowledged.
Just a bit of reference, quoting from second tutorial:
If a consumer dies (its channel is closed, connection is closed, or TCP connection is lost) without sending an ack, RabbitMQ will understand that a message wasn't processed fully and will re-queue it. If there are other consumers online at the same time, it will then quickly redeliver it to another consumer. That way you can be sure that no message is lost, even if the workers occasionally die .
114
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
