Best way to implement long running subprocess in Django?

Question

I know there are many questions similar to this one, but as far as my research has taken me, none of them answers my specific question. I hope you will take your time to help me out, as I have been struggling with this for days without finding a proper answer.

I am trying to find the best way to implement a subprocess into a Django application. To be more specific:

• The process will be run from one view (asynchronously) and handled from another.
• The process can run up to several hours.
• Multiple instances of the same process/program should be able to run at the same time.
• Other than knowing when the process is completed (or if it crashed, so it can be re-run), no communication with it is needed.

Does anyone know which way would be the best to implement this? Would any of the Python modules (such as subprocess, threads, multiprocessing, spawn) be able to achieve this or would I have to implement an external task queue such as Celery?

Answer 1

If you don't want something as complex as Celery, then you can use subprocess + nohup to start long running tasks off, dump the PID to a file (check the subprocess documentation for how to do that) and then check if the PID contained in the file is still running (using ps). And if you wanted, you could write a very small 'wrapper' script which would run the task you tell it to, and if it crashes, write a 'crashed.txt' file.

One thing to note is that you should probably run commands including the close_fds=True value to the call. (so check_call(['/usr/bin/nohup', '/tasks/do_long_job.sh'], close_fds=True) ). Why? By default, all subprocesses are given access to the parent's open file descriptors, INCLUDING ports. This means that if you need to restart your web server process, while the long process is running, that the running process will keep the port open, and you won't be able to load the server up again. You can guess how I found this out. :-)