MassTransit with RabbitMQ: messages deduplication

Question

I am using MassTransit with RabbitMQ at transport layer, and faced the need of messages deduplication.

Adding new massage to the queue should be skipped if duplicated message already queued (even if that message is processing by consumer). Duplicates could be identified by content of message for example.

Sending DoWork1, DoWork2, DoWork3 could be processed in parallel, but sending DoWork1, DoWork2, DoWork2 - duplicate should be skipped, and as far as DoWork1, DoWork2 processed same messages could be enqueued and should not be supposed as duplicates.

Solution 1: use "RabbitMQ Message Deduplication Plugin" at the exchange layer, ideal as for me, but not sure that solves described problem.

Solution 2: implement custom middleware with third party data storage.

Is there any better solution for described problem?

Thanks for help in advance!

Answer 1

The RabbitMQ deduplication plugin was designed for that purpose.

You can either de-duplicate at the exchange or at the queue. The main difference is the exchange de-duplicates a message if it has seen it previously while the queue de-duplicates it if already contains a copy of it.

When publishing a message, just set the x-deduplication-header header with a string which uniquely identifies a message (for example the MD5 hash of its body).

Using custom middleware will allow you more freedom of action at the cost of your own development.