Kinesis Lambda Consumer Minimum Batch Size

Question

I'm using AWS Lambda (node.js) as a AWS Kinesis Consumer. I can see that you can set a maximum batch size, but I'm wondering if I can set a minimum batch size. So that I can insure that each lambda will handle at least 50 (or any number) records.

I'd like to have a minimum batch size because the lambda consumer will be establishing a connection to a RDS MySQL instance and I'm trying to keep the number of concurrent connections low.

If there isn't a config capability that would set a minimum, any workaround ideas would be appreciated.

Thanks.

Answer 1

One way could be to use Kinesis Firehose, which concatenates multiple incoming records based on buffering configuration of your delivery stream.

1. Send data to Firehose - Either directly put records to Firehose Stream using their API, or attach the Firehose to your existing kinesis stream.
2. Set S3 as your Firehose's destination - This will essentially aggregate your individual records, and put them in S3 as a single object. You can specify your delimiters, and even transformation lambda functions on single records.
3. Listen for S3:PutObject - Attach your lambda to listen to the S3 bucket which receives these aggregated records from the Firehose stream.