How does Azure Service Bus identify a duplicate message?

Question

I understand that Azure Service Bus has a duplicate message detection feature which will remove messages it believes are duplicates of other messages. I'd like to use this feature to help protect against some duplicate delivery.

What I'm curious about is how the service determines two messages are actually duplicates:

• What properties of the message are considered?
• Is the content of the message considered?
• If I send two messages with the same content, but different message properties, are they considered duplicates?

Answer 1

The duplicate detection is looking at the MessageId property of the brokered message. So, if you set the message Id to something that should be unique per message coming in the duplicate detection can catch it. As far as I know only the message Id is used for detection. The contents of the message are NOT looked at, so if you have two messages sent that have the same actual content, but have different message IDs they will not be detected as duplicate.

References:

MSDN Documentation: https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-queues-topics-subscriptions

If the scenario cannot tolerate duplicate processing, then additional logic is required in the application to detect duplicates which can be achieved based upon the MessageId property of the message which will remain constant across delivery attempts. This is known as Exactly Once processing.

There is also a Brokered Message Duplication Detection code sample on WindowsAzure.com that should be exactly what you are looking for as far as proving it out.

I also quickly tested this out and sent in 5 messages to a queue with RequiresDuplicateDetection set to true, all with the exact same content but different MessageIds. I then retrieved all five messages. I then did the reverse where I had matching MessageIds but different payloads, and only one message was retrieved.

Answer 2

In my case I have to apply ScheduledEnqueueTimeUtc on top of MessageId. Because most of the time the first message already got pickup by worker, before the sub-sequence duplicate message were arrive in the Queue. By adding ScheduledEnqueueTimeUtc. We tell the Service bus to hold on the the message for some time before letting worker them up.

            var message = new BrokeredMessage(json)             {                 MessageId = GetMessageId(input, extra)             };              // Delay 30 seconds for Message to process             // So that Duplication Detection Engine has enought time to reject duplicated message             message.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddSeconds(30);