SQS delivering a message only once

Question

Let's say I have 100 threads reading from the same sqs -- is it guaranteed that each message is going to be delivered at-most once? Is it at all possible to deliver the same message more than once? I can't find any clear documentation on this issue. s

Answer 1

Update Nov 17, 2016: FIFO queueing with guaranteed-once delivery was just released today (almost 2 years after this answer was originally posted) and SQS now supports exactly-once delivery. Here are the details:

• You need to create a FIFO Queue.
• Deduplication checks for MessageDeduplicationId as an attribute of the queue message, and prevents duplicate messages from being both sent and received in the 5-minute deduplication interval following either action by checking the MessageDeduplicationId.
• If Content-based deduplication is explicitly enabled on the FIFO queue, MessageDeduplicationId will be automatically generated using a SHA-256 hash of the message body (content only, not attributes).
• If Content-based deduplication is not enabled, you must explicitly set your own arbitrary value for MessageDeduplicationId on send. Otherwise SendMessage will fail with an error.

Content-based deduplication can be enabled when creating a queue or updating the queue' attributes.

More details on MessageDeduplicationId in SendMessage and ReceiveMessage documentation.

---

After Nov 17, 2016, this still applies to standard (not-FIFO) queues:

Due to the distributed nature of standard (not-FIFO) queues in SQS, the guarantee is instead "at least" once.

From the FAQ:

Q: How many times will I receive each message?

Amazon SQS is engineered to provide “at least once” delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies.

More information on at least once delivery:

Amazon SQS stores copies of your messages on multiple servers for redundancy and high availability. On rare occasions, one of the servers storing a copy of a message might be unavailable when you receive or delete the message. If that occurs, the copy of the message will not be deleted on that unavailable server, and you might get that message copy again when you receive messages. Because of this, you must design your application to be idempotent (i.e., it must not be adversely affected if it processes the same message more than once).

If you really need to guarantee "at most once" processing in your application, you may want your application to check the unique identifiers of SQS messages and not process message IDs that you have processed before, or are currently processing.

Answer 2

You need to use FIFO queues but still you need to generate a number that guarantees no duplications of your messages, you can set MessageDeduplicationId with your generated number in the request before you send it to SQS.

Easier, You can turn on Content-Based-Deduplication for the FIFO queue and let AWS itself manage duplication for you by creating a hash based on the content of the sent message to avoid its deduplication.

• From the console, Go to SQS service, choose the queue and from Actions choose Configure queue

• Switch on Content-Based Deduplication