Background
i am studying about AWS kinesis,API gateway.
I understand that ,whenever requests hit API gateway,i can forward the data to a stream or i can choose to trigger a lambda(which will do some processing ).
Thoughts and Query
So,my thought was,if i can directly ,trigger a lambda from API gateway(When requests arrive,it is realtime),what is the advantage of having a kinesis stream(for realtime data processing)?
I could remove the streams and directly trigger lambda from API gateway(even create multiple APIs for different tasks)
Any thoughts in this scenario!
Lambda supports the following options for Kinesis event sources. Kinesis stream – The Kinesis stream to read records from. Consumer (optional) – Use a stream consumer to read from the stream over a dedicated connection. Batch size – The number of records to send to the function in each batch, up to 10,000.
AWS API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. Lambda is function as a service(FAAS) product of AWS. The combination of these two services is amazing and it is slowly replacing the traditional backend.
You can create a web API with an HTTP endpoint for your Lambda function by using Amazon API Gateway. API Gateway provides tools for creating and documenting web APIs that route HTTP requests to Lambda functions. You can secure access to your API with authentication and authorization controls.
Using API Gateway with Lambda also provides advanced capabilities, such as: Full request passthrough – Using the Lambda proxy ( AWS_PROXY ) integration type, a Lambda function can receive an entire HTTP request (instead of just the request body) and set the HTTP response (instead of just the response body).
The solution you use really depends on the data you are processing and what you'd like to do with it. More information on the data and results of your scenario could narrow down a good AWS fit. Here's a simplification of 3 options:
Kinesis Streams basically supplies a time-shifting window into a large amount of data. It takes care of storing that data long enough so you can cherry pick relevant data or perform aggregations. Your analysis of the data could be stored in a database. Kinesis Streams is a good choice when storing all of the data coming in is unnecessary and would be costly.
Kinesis Firehose provides an endpoint for you to send your data to S3, Redshift, or Elastic Search (or some combination). You can then perform your analysis on that stored data. This is a good choice if you just want your raw data to end up in a database for later processing. But, you need to pay for the storage of that data. If you only need a subset of the data or the analysis results, it would be costly.
API Gateway to Lambda allows you to process the data in real-time. The lambda could do anything you want with the data, you have the most flexibility with this solution. But you have to process each request individually, while Kinesis Streams allows you to analyze a batch of data.
It depends on frequency of client accesses and time-length of your lambda function.
The number of concurrent executions of lambda function is limited to 100. When lambda is throttled, retrying approaches are different between API Gateway and Kinesis stream.
See https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html .
You might want to check the estimation of the request rate.
In addition, keep in mind that Kinesis stream guarantees the order that data arrived in a shard.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With