Celery with Amazon SQS and S3 events

Question

I would like to use Celery to consume S3 events as delivered by Amazon on SQS. However, the S3 message format does not match what Celery expects.

How can I consume these messages with minimal hackiness? Should I write a custom serializer? Should I give up and make a custom bridge using boto or boto3?

As a sidenote, I also want to connect Celery to a different broker (RabbitMQ) for the rest of the application messaging, if that matters.

Answer 1

You're going to need to create a service that listens to the S3 notifications and then runs the appropriate celery task.

You have a variety of options - the S3 notifications go out via SQS, SNS or AWS Lambda.

In fact the simplest option may be to not use Celery at all and simply write some code to run in AWS Lambda. I haven't used this service (Lambda is relatively new) but it looks like it would mean you don't have to e.g. run a monitoring service or celery workers.