Dealing with dead letters in RabbitMQ

Question

TL;DR: I need to "replay" dead letter messages back into their original queues once I've fixed the consumer code that was originally causing the messages to be rejected.

I have configured the Dead Letter Exchange (DLX) for RabbitMQ and am successfully routing rejected messages to a dead letter queue. But now I want to look at the messages in the dead letter queue and try to decide what to do with each of them. Some (many?) of these messages should be replayed (requeued) to their original queues (available in the "x-death" headers) once the offending consumer code has been fixed. But how do I actually go about doing this? Should I write a one-off program that reads messages from the dead letter queue and allows me to specify a target queue to send them to? And what about searching the dead letter queue? What if I know that a message (let's say which is encoded in JSON) has a certain attribute that I want to search for and replay? For example, I fix a defect which I know will allow message with PacketId: 1234 to successfully process now. I could also write a one-off program for this I suppose.

I certainly can't be the first one to encounter these problems and I'm wondering if anyone else has already solved them. It seems like there should be some sort of Swiss Army Knife for this sort of thing. I did a pretty extensive search on Google and Stack Overflow but didn't really come up with much. The closest thing I could find were shovels but that doesn't really seem like the right tool for the job.

Answer 1

Should I write a one-off program that reads messages from the dead letter queue and allows me to specify a target queue to send them to?

generally speaking, yes.

you could set up a delayed re-try to resend the message back to the original queue, using a combination of the delay message exchange plugin.

but this would only automate the retries on an interval, and you may not have fixed the problem before the retries happen.

in some circumstances this is ok - like when the error is caused by an external resource being temporarily unavailable.

in your case, though, i believe your thoughts on creating an app to handle the dead letters is the best way to go, for several reasons:

• you need to search through the messages, which isn't possible RMQ
• this means you'll need a database to store the messages from the DLX/queue

because you're pulling the messages out of the DLX/queue, you'll need to ensure you get all the header info from the message so that you can re-publish to the correct queue when the time comes.

I certainly can't be the first one to encounter these problems and I'm wondering if anyone else has already solved them.

and you're not!

there are many solutions to this problem that all come down to the solution you've suggested.

some larger "service bus" implementations have this type of feature built in to them. i believe NServiceBus (or the SaaS version of it) has this built in, for example - though I'm not 100% sure of it.

if you want to look into this further, do some search for the term "poison message" - this is generally the term used for this situation. I've found a few things on google with a quick search, that may help you down the path:

• http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2013-January/025019.html
• https://web.archive.org/web/20170809194056/http://tafakari.co.ke/2014/07/rabbitmq-poison-messages/
• https://web.archive.org/web/20170809170555/http://kjnilsson.github.io/blog/2014/01/30/spread-the-poison/

hope that helps!