Celery with Amazon SQS

Question

I want to use Amazon SQS as broker backed of Celery. There’s the SQS transport implementation for Kombu, which Celery depends on. However there is not enough documentation for using it, so I cannot find how to configure SQS on Celery. Is there somebody that had succeeded to configure SQS on Celery?

Answer 1

I ran into this question several times but still wasn't entirely sure how to setup Celery to work with SQS. It turns out that it is quite easy with the latest versions of Kombu and Celery. As an alternative to the BROKER_URL syntax mentioned in another answer, you can simply set the transport, options, user, and password like so:

BROKER_TRANSPORT = 'sqs' BROKER_TRANSPORT_OPTIONS = {     'region': 'us-east-1', } BROKER_USER = AWS_ACCESS_KEY_ID BROKER_PASSWORD = AWS_SECRET_ACCESS_KEY 

This gets around a purported issue with the URL parser that doesn't allow forward slashes in your API secret, which seems to be a fairly common occurrence with AWS. Since there didn't seem to be a wealth of information out there about the topic yet, I also wrote a short blog post on the topic here:

http://www.caktusgroup.com/blog/2011/12/19/using-django-and-celery-amazon-sqs/

Answer 2

I'm using Celery 3.0 and was getting deprecation warnings when launching the worker with the BROKER_USER / BROKER_PASSWORD settings.

I took a look at the SQS URL parsing in kombo.utils.url._parse_url and it is calling urllib.unquote on the username and password elements of the URL.

So, to workaround the issue of secret keys with forward slashes, I was able to successfully use the following for the BROKER_URL:

import urllib BROKER_URL = 'sqs://%s:%s@' % (urllib.quote(AWS_ACCESS_KEY_ID, safe=''),                                urllib.quote(AWS_SECRET_ACCESS_KEY, safe='')) 

I'm not sure if access keys can ever have forward slashes in them but it doesn't hurt to quote it as well.