In Django, how do I perform server-side functions and monitor progress on client side

Question

I'm asking a basic-level question so I know how to research.

I have a python script that I am using to process large files. It runs for 1-3 minutes and has several outputs (stdout).

I want to integrate this script into django. If I run this as part of a view, it stalls the web-page waiting for the result. I lose interim progress reports (ie. "processed X records" or "Inserted Y Records").

How do I start integrating the script into the site, so I can engage the script, track progress, and show that progress and result to the user in "real-time"?

Thanks.

Answer 1

You can run your script in celery task, so your web page won't hang in there. You need to find a way to track your progress, either use database or do it in memory storage like redis. Then you can either poll the progress in your progress view or write some javascript to update the progress in the progress page.

celery and redis are very commonly used in django development. Even if it's not straight forward to setup, it's still worth to invest the time.

Answer 2

There are two primary mainstream solutions to this problem. Those being Jobtastic, and channels.

The former, Jobtastic, is potentially more mature, as it has been around and developed for some time. It uses Celery, as suggested by @ShangWang, along with a jQuery plugin which periodically polls for the current status of the job. This is more of an "old" or "traditional" web approach to this sort of job.

The latter, Channels, is a much newer library, still in development. This library would allow you to use WebSockets, or similar technologies, in order to maintain long-lived connections over which you can continually push results to the end user. Despite its relative newness and potential immaturity, this may be a more appropriate direction to research. On Dec 10, the Mozilla Open Source Support award was given to Django. This grant is being used, in part, to integrate channels into Django core. According to Andrew Godwin, the author of Channels (and Django core committer), that code will be directed at "the next major release of Django", 1.10. This means that using Channels would likely offer a clean upgrade path toward utilizing core Django functionality in future releases.