Is it possible to ensure unique messages are in a rabbitmq queue?

Question

Basically my consumers are producers as well. We get an initial dataset and it gets sent to the queue. A consumer takes an item and processes it, from that point there's 3 possibilities:

1. Data is good and gets putting a 'good' queue for storage
2. Data is bad and discarded
3. Data is not good(yet) or bad(yet) so data is broken down into smaller parts and sent back to the queue for further processing.

My problem is with step 3, because the queue grows very quickly at first its possible that a piece of data is broken down into a part thats duplicated in the queue and the consumers continue to process it and end up in a infinite loop.

I think the way to prevent against this is to prevent duplicates from going into the queue. I can't do this on the client side because over the course of an hour I may have many cores dealing with billions of data points(to have each client scan it before submitting would slow me down too much). I think this needs to be done on the server side but, like I mentioned, the data is quite large and I don't know how to efficiently ensure no duplicates.

I might be asking the impossible but thought I'd give it a shot. Any ideas would be greatly appreciated.

Answer 1

There's a plugin for rabbitmq that enables you to do this type of control with some additional headers.

You should enable the plugin and define x-deduplication-header on the message, with a hash or something that uniquely identifies the message sent, so when other message with the same header value gets into rabbitmq`s exchange it will not be routed to any queue.

See : https://github.com/noxdafox/rabbitmq-message-deduplication