Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What is a good practice to achieve the "Exactly-once delivery" behavior with Amazon SQS?

Tags:

According to the documentation:

Q: How many times will I receive each message?

Amazon SQS is engineered to provide “at least once” delivery of all messages in its queues. Although most of the time each message will be delivered to your application exactly once, you should design your system so that processing a message more than once does not create any errors or inconsistencies.

Is there any good practice to achieve the exactly-once delivery?

I was thinking about using the DynamoDB “Conditional Writes” as distributed locking mechanism but... any better idea?


Some reference to this topic:

  • At-least-once delivery (Service Behavior)
  • Exactly-once delivery (Service Behavior)
like image 452
Filippo Vitale Avatar asked Nov 21 '12 01:11

Filippo Vitale


People also ask

How can you ensure SQS messages are delivered in order?

To ensure that Amazon SQS preserves the order in which messages are sent and received, ensure that multiple senders send each message with a unique message group ID. For more information, see FIFO Queue Logic in the Amazon SQS Developer Guide.

Which Amazon SQS queue type offers maximum throughput best-effort ordering and at-least-once delivery?

SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

Is SQS exactly once?

It does not guarantee exactly-once delivery but it does guarantee exactly-once processing. What this means is that the SQS FIFO queue has to receive acknowledgement from the consumer that a message is processed before it stops returning it on future message requests.


2 Answers

FIFO queues are now available and provide ordered, exactly once out of the box.

https://aws.amazon.com/sqs/faqs/#fifo-queues

Check your region for availability.

like image 138
Sam Sippe Avatar answered Sep 28 '22 03:09

Sam Sippe


The best solution really depends on exactly how critical it is that you not perform the action suggested in the message more than once. For some actions such as deleting a file or resizing an image it doesn't really matter if it happens twice, so it is fine to do nothing. When it is more critical to not do the work a second time I use an identifier for each message (generated by the sender) and the receiver tracks dups by marking the ids as seen in memchachd. Fine for many things, but probably not if life or money depends on it, especially if there a multiple consumers.

Conditional writes sound like a clever solution, but it has me wondering if perhaps AWS isn't such a great solution for your problem if you need a bullet proof exactly-once solution.

like image 42
Jeff Avatar answered Sep 28 '22 01:09

Jeff