Menu
I have a Lambda function to which multiple DynamoDB streams are configured as event sources and this is a part of a bigger pipeline. While doing my checks, I found some missing data in one of the downstream components. I want to write a simpler Lambda function which is configured as an event source to one of the earlier mentioned DynamoDB streams. This would cause one of my DynamoDB streams to have two Lambda functions reading from it. I was wondering if this is OK? Are both Lamdba functions guaranteed to receive all records placed in the stream and are there any resource (Read/Write throughput) limits I need to be aware of. Couldn't find any relevant documentation for this on the AWS website, but I did find this regarding processing of shards
To access a stream and process the stream records within, you must do the following:
- Determine the unique Amazon Resource Name (ARN) of the stream that you want to access.
- Determine which shard(s) in the stream contain the stream records that you are interested in.
- Access the shard(s) and retrieve the stream records that you want.
Note No more than 2 processes at most should be reading from the same Streams shard at the same time. Having more than 2 readers per shard may result in throttling.
Not sure how the above relates to cases where Streams are configured as Event sources to Lambdas as opposed to manually reading from a Stream using the API.
890
You can associate a DynamoDB stream with multiple Lambda functions, and associate the same Lambda function with multiple streams. However, the Lambda functions will share the read throughput for the stream they share. You can get the list of event source mappings by running the following command.
For DynamoDB streams, these limits are even more strict -- AWS recommends to have no more than 2 consumers reading from a DynamoDB stream shard. If you had more than 2 consumers, as in our example from Part I of this blog post, you'll experience throttling.
Serverless applications usually consist of multiple Lambda functions. Each Lambda function can use only one runtime but you can use multiple runtimes across multiple functions. This enables you to choose the best runtime for the task of the function.
With DynamoDB Streams, you can trigger a Lambda function to perform additional work each time a DynamoDB table is updated. Lambda reads records from the stream and invokes your function synchronously with an event that contains stream records.
You can have multiple Lambdas using the same stream as an event source. They will not interfere with each other. But as the documentation says: "Note No more than 2 processes at most should be reading from the same Streams shard at the same time. Having more than 2 readers per shard may result in throttling." So if you heavily utilize your streams you should not have more than two Lambdas connected to them.
117
This AWS Blog post https://aws.amazon.com/de/blogs/database/how-to-perform-ordered-data-replication-between-applications-by-using-amazon-dynamodb-streams/ suggest that you attach only one Lambda to the DDB stream and use a fan out pattern for parallel processing. This will help you processing the DDB items in order.
35
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
