Kinesis stream / shard - multiple consumers

Question

I have already read some questions about kinesis shard and multiple consumers but I still don't understand how it works.

My use case: I have a kinesis stream with just one shard. I would like to consume this shard using different lambda function, each of them independently. It's like that each lambda function will have it's own shard iterator.

Is it possible? Set multiple lambda consumers ( stream based) reading from the same stream/shard?

Answer 1

Hey Mr Magalhaes I believe the following picture should answer some of your questions.

So to clarify you can set multiple lambdas as consumers on a kinesis stream, but the Lambdas will block each other on processing. If your stream has only one shard it will only have one concurrent Lambda.

Answer 2

If you have one kinesis stream, you can connect as many lambda functions as you want through an event source mapping.

All functions will run simultaneously and fully independent of each other and will constantly be invoked if new records arrive in the stream. The number of shards does not matter.

Answer 3

For a single lambda function: "For Lambda functions that process Kinesis or DynamoDB streams the number of shards is the unit of concurrency. If your stream has 100 active shards, there will be at most 100 Lambda function invocations running concurrently. This is because Lambda processes each shard’s events in sequence." [https://docs.aws.amazon.com/lambda/latest/dg/scaling.html]

But there is no limit on how many different lambda consumers you want to attach with kinesis.

Answer 4

Yes, no problem with this !

The number of shards doesn't limit the number of consumers a stream can have. In you case, it will just limit the number of concurrent invocations of each lambda. This means that for each consumers, you can only have the number of shards of concurrent executions.

Seethis doc for more details.