Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

If a AWS Lambda function has event sources from multiple Kinesis streams, will the batch of incoming records be from a single Kinesis stream or a mix?

The title might be a bit confusing. I'll try my best to make it clearer.

Suppose I have a AWS Lambda function that has two different Kinesis streams A and B as input event sources.

So, for the below, since a KinesisEvent instance contains a batch of records, will the batch contain records from a single stream, or essentially it contain records from both streams A and B?

public class ProcessKinesisEvents {
    public void recordHandler(KinesisEvent event, Context context) {
        ...
    }
}
like image 540
Gordon Tai Avatar asked Sep 05 '15 07:09

Gordon Tai


People also ask

How does Lambda work with Kinesis?

For standard iterators, Lambda polls each shard in your Kinesis stream for records at a base rate of once per second. When more records are available, Lambda keeps processing batches until the function catches up with the stream. The event source mapping shares read throughput with other consumers of the shard.

Can one Lambda function have multiple triggers?

Your function can have multiple triggers. Each trigger acts as a client invoking your function independently. Each event that Lambda passes to your function has data from only one client or trigger.

How many put records per second does Amazon Kinesis data streams support?

It serves as a base throughput unit of a Kinesis data stream. A shard supports 1 MB/second and 1,000 records per second for writes and 2 MB/second for reads.

How does a Kinesis data streams distribute data to different shards?

Kinesis Data Streams segregates the data records belonging to a stream into multiple shards. It uses the partition key that is associated with each data record to determine which shard a given data record belongs to. Partition keys are Unicode strings, with a maximum length limit of 256 characters for each key.


1 Answers

Each mapping between an AWS Kinesis stream and an AWS Lambda function is a dedicated entity resulting from a call to CreateEventSourceMapping and comprised of the EventSourceArn and the FunctionName, for which you also specify a dedicated Batch Size accordingly:

POST /2015-03-31/event-source-mappings/ HTTP/1.1
Content-type: application/json

{
    "BatchSize": number,
    "Enabled": boolean,
    "EventSourceArn": "string",
    "FunctionName": "string",
    "StartingPosition": "string"
}

Consequently, the batches you will receive are constrained to the single event source that constitutes the resp. mapping, and each other event source will yield a separate invocation of your Lambda function accordingly, so everything is properly isolated.

like image 142
Steffen Opel Avatar answered Sep 21 '22 10:09

Steffen Opel