Processing AWS SQS messages with separate Lambda at a time

Question

Like the title suggests, I have a scenario that I would like to explore but do not know how to go about it.

I have a lambda function processCSVFile. I also have a SQS queue that at a set time everyday, it gets populated with link of csv files from S3, let's say about 2000 messages. Now I want to process 25 messages at a time once the SQS queue has the messages.

The scenario I am looking for is to process 25 messages concurrently, I want the 25 messages to be processed by 25 lambda invocations separately. I thought I could use SendMessageBatch function in SQS but this only delivers messages to the queue, it does not seem to apply to my use case.

My question is, am I able to perform the action explained above and if it is possible, what documentation or use cases can explain what I am looking for. Also, if this use case is impossible, what do you recommend as an alternative way to do the processing I want done concurrently.

Answer 1

To process 25 messages from Amazon SQS with 25 concurrent Lambda functions (1 message per running Lambda function), you would need:

• A maximum concurrency of 25 configured for the Lambda function (otherwise it might go higher than this when more messages are available)
• A batch size of 1 configured on the Lambda trigger so that SQS only passes it one message at a time

See:

• AWS Lambda Function Scaling (Maximum concurrency)
• Configuring a Queue as an Event Source (Batch size)