AWS Lambda SQS Trigger Throttle/Limit

Question

I have an AWS SQS queue that I'm going to setup with a Lambda function trigger to run the Lambda function for every item that gets added to the queue to do some processing work.

One step of processing is going to be hitting an API endpoint to get some data back for every item added to the queue, then storing that away in an DynamoDB table. In order to manage costs, and stay in control of this process I'm wanting to throttle the amount of times this Lambda function gets called in a certain period of time.

For example, I want this function to be run a maximum of 100 times per day, in order to not overwhelm the DynamoDB table capacity or the API endpoint. It would also be nice to throttle it, to where it will only have a maximum of 5 concurrent actions being run at once, with a 1 second delay between running again. This type of control would allow me to directly map the function to the DynamoDB table limits to ensure I'm not going over capacity, and to I comply with any API rate limits.

I have looked into AWS Lambda Managing Concurrency. Specifically the "Function Level Concurrent Execution Limit" section. But this section doesn't seem to address the 100 times per day limit, or the 1 second delay between running the next function.

I also know that I could just limit it on the other side, by limiting the number of items I put in the queue at a time (or per day). But because of how I'm planning this system, that would add a lot of complexity that I would love to try to avoid.

Is there a way using AWS SQS and Lambda to achieve this and limit these Lambda functions?

Answer 1

I don't think there is a way to limit the number of Lambda invocations when using SQS as an event source to 100 a day.

An alternative method would be to have a scheduled event trigger a Lambda, which polls the SQS queue. While this could increase the lag between data going from SQS to DynamoDB, you'd have a lot more control for the number and rate of writes to DynamoDB.

To manage the 100 count, you could either hold the daily write count state externally (e.g. S3) or divide 100 by the number of scheduled invocations, e.g. 10 invocations of a Lambda which stops after processing 10 messages. Either way, you will need to ensure your Lambda does not time out half way through processing a message.

As a single invocation of Lambda will be making multiple writes to DynamoDB you can manage how many writes are made per sec, or build in retries.