Could gunicorn cause an issue with exscript/paramiko?

Question

I have a Django project running behind Nginx, and Gunicorn. One of the applications interacts with network devices using Exscript, which is in turn using Paramiko. Some devices do not work properly when they are behind Gunicorn.

The same exact code will work fine from within the django-admin shell. It will also work if I launch the built in django server, but I still get the error if I bypass nginx, and connect directly to Gunicorn.

I tried moving the functionality to a celery task, it had the same exact problem, but only behind Gunicorn.

I wrote a script using django-extensions that works from the command line, but will fail if called via subprocess. But only behind Gunicorn.

The devices that are failing all seem to be F5 LTMs, and it looks like the buffer on the exscript object is being modified somehow. If I had to guess I would say that Gunicorn, and Exscript/Paramiko are somehow stepping on each others memory, or perhaps Gunicorn is setting some environment variable that Exscript is picking up on.

In any case I am thoroughly stumped, and would appreciate any guidance on how to troubleshooot this.

Answer 1

Total guess, but perhaps this will be helpful in debugging. Basically, ensure you've removed all output buffering, which can often be hiding what is really happening when layering multiple big frameworks (like you are doing here).

Ensure that you disable all output buffering in Python, both for your foreground webserver process and for any worker processes (setting PYTHONUNBUFFERED is an easy way to ensure that none of your python scripts have buffering, at least on the standard library functions).

The terminal can also introduce buffers that make debugging exceptionally hard. Consider switching your command to stdbuf -o0 -e0 your command to disable buffers on stdout and stderr (your command could still re-enable them, but most programs do not).