Run a lambda on every DynamoDb entry on schedule?

Question

Is there a way to run a Lambda on every DynamoDb table record?

I have a Dynamo table with name, last name, email and a Lambda that takes name, last name, email as parameters. I am trying to configure the environment such that, every day, the Lambda runs automatically for every value it finds within Dynamo; can't do all the records in one Lambda as it won't scale (will timeout once more users are added).

I currently have a CloudWatch rule set up that triggers the lambda on schedule but I had to manually add the parameters to the trigger from Dynamo - It's not automatic and not dynamic/not connected to dynamo.

--

Another option would be to run a lambda every time a DynamoDb record is updated... I could update all the records weekly and then upon updating them the Lambda would be triggered but I don't know if that's possible either.

Some more insight on either one of these approaches would be appreciated!

Answer 1

Is there a way to run a Lambda on every DynamoDb table record?

For your specific case where all you want to do is process each row of a DynamoDB table in a scalable fashion, I'd try going with a Lambda -> SQS -> Lambdas fanout like this:

1. Set up a CloudWatch Events Rule that triggers on a schedule. Have this trigger a dispatch Lambda function.

2. The dispatch Lambda function's job is to read all of the entries in your DynamoDB table and write messages to a jobs SQS queue, one per DynamoDB item.

3. Create a worker Lambda function that does whatever you want it to do with any given item from your DynamoDB table.

4. Connect the worker Lambda to the jobs SQS queue so that an instance of it will dispatch whenever something is put on the queue.

Answer 2

Since the limiting factor is lambda timeouts, run multiple lambdas using step functions. Perform a paginated scan of the table; each lambda will return the LastEvaluatedKey and pass it to the next invocation for the next page.