Django-utils @async decorator and gunicorn not working

Question

I use djutils @async decorator for calling a function asynchronously. This works well when i start my server with the standard ./manage.py runserver command. But when i run my django app under gunicorn and i call the function with the @async decorator nothing happens at all...

I looked into the code of the djutils async decorator and it adds and object to a python-standard Queue, and also spawning one or more threads to consume the queue.

anyone an idea or hint where i should look next, to get my @async functions running in gunicorn? Is gunicornn changing the standard implementation of python thread implementation?

Answer 1

Python threads are subject to interference from the global interpreter lock and Gunicorn is most likely monkey-patching the default threading behavior. So you could try changing its Worker Type. If your Gunicorn is configured to use async workers with Gevent, then it may monkey patch Thread because Gevent can do that, replacing the thread class with a "green" thread that's cooperative.

Personally I hate "do it differently" answers, so sorry for giving one. Don't spawn threads or fork processes from inside your web server's process. Just offload that work to a background queue.

Building your own background queue is easy if you have simple needs. Have a table in your database that logs pending and completed jobs, in combination with a cron script that executes every 5 minutes or so and work from there.

And if you want something stable and that can scale, Celery is pretty good.