Consume SQS messages using AWS lambda function

Question

I have 2 FIFO SQS queues which receives JSON messages that are to be indexed to elasticsearch. One queue is constantly adding delta changes to the database and adding them to the queue. The second queue is used for database re-indexing i.e. the entire 50Tb if data is to be indexing every couple of months (where everything is added to the queue). I have a lambda function that consumes the messages from the queues and places them into the appropriate queue (either the active index or the indexing being rebuilt).

How should I trigger the lambda function to best process the backlog of messages in SQS so it process both queues as quickly as possible?

A constraint I have is that the queue items need to be processed in order. If the lambda function could be run indefinitely without the 5 minute limit I could keep running one function that constantly processes messages.

Answer 1

Instead of pushing your messages directly into SQS you could publish the messages to a SNS Topic with 2 Subscriber registered.

1. Subscriber: SQS
2. Subscriber: Lambda Function

Has the benefit that your Lambda is invoked at the same time as the message is stored in SQS.