Python/Django AMQP?

Question

I have a Django web application which creates and saves jobs in a database. For every job I have to send email and export data to another system. I want to make asynchronous.

What is the recommended way to do tasks like this?

• It has to be reliable (no jobs missed if the service is stopped accidentally).
• There has to be easy way to find logs if something goes wrong.

• I want to be able to easily add more workers that will process the jobs.

• Maybe it will be better if the job queue is the same database table so I can easily cancel jobs by simply changing their status in database?

• Maybe it will be better if the solution is somehow related to the Django framework?

Answer 1

I would look into using http://celeryproject.org/ and in your case you could probably use https://github.com/ask/django-celery to simplify integration with your existing application although using celery yourself within your django project should be pretty straightforward as well.

Celery provides a function/class based abstraction for tasks that should be run asynchronously on worker machines. You can have multiple worker machines and add workers dynamically. The whole thing is powered by RabbitMQ (www.rabbitmq.com) so your django app will put tasks into the queue by basically doing something like:

from mycelerytasks import send_email

...

deferred_result = send_email.apply_async(*args, **kwargs)

If you want to wait for the task to complete you could do deferred_result.wait() but in your case you'd probably just discard it and let your view finish so the user isn't waiting around for an answer.

Answer 2

I'm not sure this is what you want, but I handle this kind of thing with a periodic cron job on my linux server, running a python script. (A scheduled task would be equivalent on windows)

You can gain access to your project's modules and settings in your python script by manipulating the sys.path list, like so:

import sys
import os
sys.path.append( PATH_TO_PROJECT_DIR )
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

This is perhaps a bit of a raw solution in that it's up to you to sort out the logging, job queue etc..., but works quite well for simple house keeping things like sending emails and is easy to implement.