Long running tasks with Django

Question

My goal is to create an application that will be able to do long-lasting mainly system tasks, such as:

• checking out code from the repositories,
• copying directories between various localizations,
• etc.

The problem is I need to prepare it somehow independently from the web browser. I mean that for example after starting the checkout/copy action, closing the web browser will not interrupt the action. So after going back to that site I can see that the copying goes on or another action started when the browser was closed...

I was searching through various tools, like RabbitMQ + Celery, Twisted, Pyro, XML-RPC but I don't know if any of these will be suitable for me. Has anyone encountered similar needs when creating Django app? Please let me know if there are any methods/packages that I should know. Code samples also will be more than welcome!

Thank you in advance for your suggestions!

(And sorry for my bad English. I'm working on it.)

Answer 1

Basically you need to have a process that runs outside of the request. The absolute simplest way to do this (on a Unix-like operating system, at least) is to fork():

if os.fork() == 0:
    do_long_thing()
    sys.exit(0)
… continue with request …

This has some downsides, though (ex, if the server crashes, the “long thing” will be lost)… Which is where, ex, Celery can come in handy. It will keep track of the jobs that need to be done, the results of jobs (success/failure/whatever) and make it easy to run the jobs on other machines.

Using Celery with a Redis backend (see Kombu's Redis transport) is very simple, so I would recommend looking there first.

Answer 2

You might need to have a process outside the request / response cycle. If that is the case, Celery with a Redis backend is what I would suggest looking into, as that integrates nicely with Django (as David Wolever suggested).

Another option is to create Django management commands, and then use cron to execute them at scheduled intervals.