How to handle S3 events inside a Kubernetes Cluster?

Question

I have a Kubernetes cluster that runs on AWS EKS, now I want to handle S3 object creation events in a pod, like I would with AWS Lambda.

How can I handle S3 events from inside a Kubernetes cluster?

Answer 1

S3 events can only be sent to SQS, SNS or Lambda functions. So any S3 -> K8s integration will have to use those event destinations. I can think of couple of options to trigger K8s Jobs using S3 events (I'm assuming the K8s cluster is EKS)

Option 1:

1. S3 events are sent to a SQS queue
2. K8s Job polling the SQS queue -- the Job will have a ServiceAccount mapped to an IAM role which can read from SQS

Option 2:

1. S3 events trigger a Lambda function

2. This Lambda function uses container image which has kubectl/helm installed.

3. Lambda function's IAM role will be mapped to a K8s Role

4. Lambda function runs some code/script that creates following when invoked (using kubectl/helm)

   a. A ConfigMap with the event information is

   b. A Job that takes the ConfigMap name/label as a parameter or environment variable