AWS FIFO Queues with lambda – one concurrent lambda per message group

Question

I have set up a FIFO queue and I'd like to partition messages by a message group ID.

As an event trigger, I'm using a lambda function.

Now my question: Is it possible to allow only one concurrent lambda function invocation for every message group?

So, if I have Group A and Group B, and each message group has 20 messages, every message group passes one message at a time (I have set up the batch size of 1) to a lambda function and only passes the next one as soon as the previous message processing has finished.

Is that possible using FIFO queues and lambda or do I need to look into other services to allow that?

Note: I've looking into Kinesis with separate shards already, but because there'll be many message groups, the total cost of the shards would be way too much.

Thank you!

Answer 1

It should work the way you want out of the box.

New for AWS Lambda – SQS FIFO as an event source:

In SQS FIFO queues, using more than one MessageGroupId enables Lambda to scale up and process more items in the queue using a greater concurrency limit. Total concurrency is equal to or less than the number of unique MessageGroupIds in the SQS FIFO queue.