Django + gunicorn + nginx: 502 bad gateway but sometimes only?

Question

I recently decided to just give up on apache2 + mod_wsgi and give a try to gunicorn + nginx to run my Django App.

I followed those tutorials with no issue at all, on a clean Ubuntu 16.04 install:

• https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04
• https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-16-04#create-a-gunicorn-systemd-service-file

I thought that everything would be fine, but when browsing my website, I noticed that from time to time a few pages did not display at all and just show a 502 Bad Gateway error. Looking into the error/var/log/nginx/error.log, I only have errors like the following:

upstream prematurely closed connection while reading response header from upstream

I have no clue on what it implies:

• why would it be my Django install, if I have no issues when running on localhost for instance?
• also: sometimes all pages work and display just fine; sometimes the sames pages will throw a 502 error :(
• how could I know what's wrong with my gunicorn install, if the issue comes from there? How to debug?

For you information:

---

/etc/systemd/system/gunicorn.socket

[Unit]
Description=gunicorn socket

[Socket]
ListenStream=/run/gunicorn.sock

[Install]
WantedBy=sockets.target

---

/etc/systemd/system/gunicorn.service

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target

[Service]
User=myself
Group=www-data
WorkingDirectory=/home/myself/django/myproject
ExecStart=/home/myself/django/myproject/venvprod/bin/gunicorn \
          --access-logfile - \
          --bind unix:/run/gunicorn.sock \
          myproject.wsgi:application

[Install]
WantedBy=multi-user.target

---

/etc/nginx/sites-enabled/myproject

server {
    server_name mysite.fr mysite.com;
    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
        root /home/myself/django/myproject;
    }

    location / {
        include proxy_params;
        proxy_pass http://unix:/run/gunicorn.sock;
    }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/mysite.fr/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/mysite.fr/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host = mysite.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    if ($host = mysite.fr) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    server_name mysite.fr mysite.com;
    return 404; # managed by Certbot
}

---

If anything else is needed, please just ask! My real issue is that I have no idea how to find the problem. What tool could I use?

Thanks in advance for your help.

---

EDIT 1

I can confirm I do not have any error coming from my Django code. I logged everything as follows:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': '/home/krazymax/debug-m2g.log',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'django.template': {
            'handlers': ['file'],
            'level': 'INFO',
            'propagate': True,
        },
    },
}

and I do not have any error logged in while browsing my website (with pages that always display, other that never display and some that sometimes display, sometimes don't display).

---

EDIT 2

I tried to follow this tutorial, ie without using an intermediate sock file. No success, even if when executing myvenv/bin/gunicorn -c myvenvprod/gunicorn_config.py myproject.wsgi I am able to access my website (and not otherwise): I am still having my pages being displayed (or not) randomly. I really have no clue and it's official, this random behavior is driving me crazy!

Answer 1

This error usually means that there is not a "connection" between Nginx and Gunicorn, the problem is the socket file.

Run "/home/myself/django/myproject/venvprod/bin/gunicorn myproject.wsgi -b 0.0.0.0:8000" and see the output. This execute the gunicorn server with your Django process without demonize it.

Maybe there is an error in your Django code and the socket is not created correctly. While Gunicorn is open try also to visit YOUR_IP:8888 (es. 52.45.241.21:8888) from your browser: do you see the website?

If you see some Python errors you should first debug your project with "manage.py runserver 0.0.0.0:8000".

The 0.0.0.0:8000 after runserver allows your site to be visited form an external browser (like -b 0.0.0.0:8000 option in Gunicorn)

Remember: before follow any tutorial on a Django project deploy check that the project works correctly on the machine using runserver

EDIT: Try also an easier tutorial like this: https://gist.github.com/ravidsrk/8431321

Hope this hepls! :)