Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How does AWS FIFO SQS deduplication ID work?

Tags:

I'm trying to use the AWS SQS FIFO service together with an Elastic Beanstalk worker environment.

Let's say I send a message with MessageDeduplicationId test, if I continue sending this exact message in the next 5 minutes, the message will be ignored, correct?

What happens if I send a message with MessageDeduplicationId test , the consumer processes the message and deletes it, then, in about 1 minute, I send the exact same message again. Will this message be ignored?

My question is, does deduplication occur as long as the same MessageDeduplicationId is still in queue/flight? Or is the id banner forever, no other message with the same id can be sent.

Thanks.

like image 558
tehmaestro Avatar asked Dec 04 '17 17:12

tehmaestro


People also ask

What is deduplication ID in AWS SQS?

PDFRSS. The message deduplication ID is the token used for deduplication of sent messages. If a message with a particular message deduplication ID is sent successfully, any messages sent with the same message deduplication ID are accepted successfully but aren't delivered during the 5-minute deduplication interval.

What is deduplication in FIFO queue?

Amazon SNS FIFO topics and Amazon SQS FIFO queues support message deduplication, which provides exactly-once message delivery and processing as long as the following conditions are met: The subscribed SQS FIFO queue exists and has permissions that allow the Amazon SNS service principal to deliver messages to the queue.

How does the AWS FIFO SQS queue send messages only once without any duplicates?

FIFO queues provide exactly-once processing, which means that each message is delivered once and remains available until a consumer processes it and deletes it. Duplicates are not introduced into the queue.

When content based deduplication is enabled the message deduplication ID is optional?

If you enabled content-based deduplication for the queue, the message deduplication ID isn't required. For more information, see FIFO delivery logic. (Optional) For a standard queue, you can enter a value for Delivery delay and choose the units. For example, enter 60 and choose seconds.


Video Answer


2 Answers

What happens if I send message with MessageDeduplicationId test , the consumer processes the message and the deletes it, then, in about 1 minute, I send the exact same message again. Will this message be ignored?

Answer seems to be: Yes

Amazon SQS continues to keep track of the message deduplication ID even after the message is received and deleted.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagededuplicationid-property.html

like image 66
antak Avatar answered Oct 03 '22 14:10

antak


I continue sending this exact messages in the next 5 minutes, the message will be ignored, correct?

Correct. Any longer than 5 minutes, though, and it will be accepted again and delivered again and then ignored for another 5 minutes.

like image 44
Michael - sqlbot Avatar answered Oct 03 '22 14:10

Michael - sqlbot