Increasing DynamoDB Stream + Lambda throughput

Question

I have a DynamoDB Stream that triggers a Lambda function. I'm noticing that bursts of a thousand of writes to the DynamoDB table can take many minutes (longest I have seen is 30 minutes) to all be processed by Lambda. The average duration for each Lambda invocation with batch size of 3 is around 2 seconds. These Lambdas perform I/O heavy tasks, so a small batch size and a higher number of parallel invocations is advantageous. However, the parallelism of these Lambdas is pegged to the number of DynamoDB Stream shards, but I cannot find a method for scaling the number of shards.

Is there any way to increase the throughput of these Lambdas beyond using a bigger batch size and more optimized code?

Answer 1

Each stream shard is associated with a partition in DynamoDB. If you increase the throughput on your table so much that you cause your partitions to split, then, you will get more shards. With more shards the number of Lambda functions that run in parallel will increase.

Answer 2

I don't see much configuration options either.

You could decouple your processing. If your change records aren't too large your incoming Lambda could just split them up into several smaller SNS messages. Each of those smaller SNS messages could than trigger a Lambda doing the actual processing. If the changes are larger you could use SQS or S3 and trigger Lambda processing for new messages via SNS or directly for files.