Executing a python script using subprocess.Popen() in a django view

Question

I've looked around a bit but I can't seem to solve this problem I have. I'd like to execute a python script within a view of my django app. I've placed the code I'd like to execute inside a django management command so it can be accessed via command line python manage.py command-name. I then tried to run this command using subprocess.Popen("python manage.py command-name",shell=True).

However, this command could take some time to execute so I'd like the view to continue and allow the script to execute in the background. Using subprocess.Popen alone seems to cause the view to hang until the script has finished, so I tried using a thread (following another SA question):

class SubprocessThread(threading.Thread):
def __init__(self, c):
    self.command = c
    self.stdout = None
    self.stderr = None
    threading.Thread.__init__(self)

def run(self):
    p = subprocess.Popen(self.command,
                         shell=True,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE)

    self.stdout, self.stderr = p.communicate()

and then executing it:

t = SubprocessThread("python manage.py command-name")
t.setDaemon(True)
t.start()
t.join()

However, the view still hangs: the cursor has a busy symbol and the AJAX on the page does not load. Otherwise the page's html seems to load fine and commands in the view after the thread call appear to finish normally (before the script finishes). Can someone please help me? I'd like the script to execute and do its own thing without holding up the view or AJAX calls on the page.

Answer 1

Maybe you should use celery

Celery is a task queue/job queue based on distributed message passing. It is focused on real-time operation