SQS fifo queues not ensuring single time delivery when used as lambda trigger

Question

I have a fifo queue for which it acts trigger to a lambda and there are no other consumers for this fifo queue.

I want my lambda to not receive any duplicates, for that I am making sure that my messages have uniqueid and content based deduplication is enabled such that there are no duplicates. But as the documentation says

Amazon SQS FIFO queues ensure that the order of processing follows the message order within a message group. However, it does not guarantee only once delivery when used as a Lambda trigger. If only once delivery is important in your serverless application, it’s recommended to make your function idempotent. You could achieve this by tracking a unique attribute of the message using a scalable, low-latency control database like Amazon DynamoDB.

Does this mean I will receive duplicates to my lambda where it is the only consumer of the fifo queue even though deduplication is enabled?

Answer 1

There are two things in play here that you seem to be mixing together.

• On one hand, there's the delivery model of SQS. With SQS FIFO queues, you correctly noted that it enables exactly-once delivery.

• On the other hand, there's the execution model of Lambda functions. The execution model is at-least-once. This is independent of any concurrency settings, by the way.

There are multiple reasons why Lambda may execute a function more than once. Most notably, and independently of SQS FIFO queues, Lambda has built-in retry. Depending on what kind of error may happen, and what kind of externally observable side effects your Lambda code has, you may see your code run more than once for a single actual invocation.

That said, there are still other reasons why a message sent to an SQS FIFO queue may end up being processed by Lambda more than once. For example, if your Lambda function takes longer than the VisibilityTimeout settings of the queue or message to finish processing the entire batch it received, then all those messages are going to become Visible again in the queue, and another invocation of your Lambda function will (certainly, not just "probably") receive those messages again.

So the bottom line is: you need to write idempotent code in Lambda - not because of SQS FIFO (it does correctly allow for prevention of duplicate deliveries), but because of Lambda retries and th potential for the messages to become visible again in the queue due to slow processing (it's always at-least-once execution model).

Answer 2

Even you have a single lambda function as consumer, depends on the concurrency setting on your lambda function, there can be multiple invocations. Which means at a given time, multiple invocations can pick messages in your SQS queue.

Simple workaround for this is to set the lambda function's concurrency to 1, so that at a given time, only one invocation is allowed. So the messages in the queue will be processed in order.

(However this will leads to a bottleneck if you have a large number of messages in the queue to be processed.)