Is it possible to integrate AWS Lambda with Apache Kafka ? I want to put a consumer in a lambda function. When a consumer receive a message the lambda function execute.
AWS Lambda now supports Amazon Managed Streaming for Apache Kafka (Amazon MSK) as an event source, giving customers more choices to build serverless applications with streaming data. Customers can build Apache Kafka consumer applications with Lambda functions without needing to worry about infrastructure management.
Records within different topic partitions, though, can be processed in parallel. If configured, the response from AWS Lambda can be written to a Kafka topic.
Learn more about Kafka on AWSAWS also offers Amazon MSK, the most compatible, available, and secure fully managed service for Apache Kafka, enabling customers to populate data lakes, stream changes to and from databases, and power machine learning and analytics applications.
Continuing the point by Arafat. We have successfully built an infrastructure to consume from Kafka using AWS Lambdas. Here are some gotcha's:
context
object in the Lambda and give yourself some wiggle room to do something with the buffer you populated in your consumer which might not be read to a file unless you call close()
.We are using Apache Airflow for scheduling. I hear cloudwatch can do that too.
Here is AWS article on scheduled lambdas.
Given your Kafka installation will be running in a VPC, best practise is to configure your Lambda to run within the VPC as well - this will simplify the security group configuration for the EC2 instances running Kafka.
Here is the AWS blog article on configuring Lambdas to run in a VPC.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With