Is there a way to increase contentBasedDeduplication
interval in SQS FIFO Queues? In the developer guide they mention
If a message with a particular message deduplication ID is sent successfully, any messages sent with the same message deduplication ID are accepted successfully but aren't delivered during the 5-minute deduplication interval.
However in the API reference they mention:
When ContentBasedDeduplication is in effect, messages with identical content sent within the deduplication interval are treated as duplicates and only one copy of the message is delivered.
When I try to create a FIFO queue I do not see any option to change the deduplication interval. Is that fixed?
Amazon SQS has a deduplication interval of 5 minutes. Retrying SendMessage requests after the deduplication interval expires can introduce duplicate messages into the queue. For example, a mobile device in a car sends messages whose order is important.
Unlike standard queues, FIFO queues don't introduce duplicate messages. FIFO queues help you avoid sending duplicates to a queue. If you retry the SendMessage action within the 5-minute deduplication interval, Amazon SQS doesn't introduce any duplicates into the queue.
FIFO throughput limit – Specifies whether the throughput quota on messages in the FIFO queue is set at the queue or message group level.
Q: Do Amazon SQS FIFO queues support multiple consumers? By design, Amazon SQS FIFO queues don't serve messages from the same message group to more than one consumer at a time.
In FIFO queues, the Deduplication Interval
is fixed for 5 mins. Currently there is NO way to increase the deduplication time window.
This concern has been raised in one of the AWS Forums in 2017, but still it seems like this is not available as a feature.
Additional Note:
The purpose of having a deduplication Interval
is to avoid processing the same message twice within the given 5 minutes. If you want to increase it, it would seem like an overhead on SQS because it needs to keep track of previous messages for a longer period.
I suggest having a filtering layer between your program and SQS, to remove duplicates, according to your required logic. Same with if you want to decrease it.
Hope this helps.
There are no ways to change the deduplication time. Deduplication will monitor the message even if it is deleted from the queue. For FIFO queues, the Deduplication Interval is fixed for 5 mins.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With