Menu
I'm using AWS Lambda (node.js) as a AWS Kinesis Consumer. I can see that you can set a maximum batch size, but I'm wondering if I can set a minimum batch size. So that I can insure that each lambda will handle at least 50 (or any number) records.
I'd like to have a minimum batch size because the lambda consumer will be establishing a connection to a RDS MySQL instance and I'm trying to keep the number of concurrent connections low.
If there isn't a config capability that would set a minimum, any workaround ideas would be appreciated.
Thanks.
881
Batch size – The number of records to send to the function in each batch, up to 10,000. Lambda passes all of the records in the batch to the function in a single call, as long as the total size of the events doesn't exceed the payload limit for synchronous invocation (6 MB).
Batch size – The number of records to send to the function in each batch. For a standard queue, this can be up to 10,000 records. For a FIFO queue, the maximum is 10. For a batch size over 10, you must also set the MaximumBatchingWindowInSeconds parameter to at least 1 second.
You can register up to 20 consumers per data stream. A given consumer can only be registered with one data stream at a time. Only 5 consumers can be created simultaneously. In other words, you cannot have more than 5 consumers in a CREATING status at the same time.
Kafka requires more engineering hours for implementation and maintenance leading to a higher total cost of ownership (TCO). As an AWS cloud-native service, Kinesis supports a pay-as-you-go model leading to lower costs to achieve the same outcome.
One way could be to use Kinesis Firehose, which concatenates multiple incoming records based on buffering configuration of your delivery stream.
175
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
