websockets proxied by nginx to gunicorn over https giving 400 (bad request)

Question

I am having trouble establishing a websocket in my Flask web application.

On the client side, I am emitting a "ping" websocket event every second to the server. In the browser console, I see the following error each second

POST https://example.com/socket.io/?EIO=3&transport=polling&t=LOkVYzQ&sid=88b5202cf38f40879ddfc6ce36322233 400 (BAD REQUEST)

GET https://example.com/socket.io/?EIO=3&transport=polling&t=LOkVZLN&sid=5a355bbccb6f4f05bd46379066876955 400 (BAD REQUEST)

WebSocket connection to 'wss://example.com/socket.io/?EIO=3&transport=websocket&sid=5a355bbccb6f4f05bd46379066876955' failed: WebSocket is closed before the connection is established.

I have the following nginx.conf

server {
    listen 80;
    server_name example.com www.example.com;
    return 301 https://$server_name$request_uri;
}

upstream app_server {

    # for UNIX domain socket setups
    server unix:/pathtowebapp/gunicorn.sock fail_timeout=0;

}

server {

    listen 443 ssl;

    server_name example.com www.example.com;

    keepalive_timeout 5;

    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';

    charset     utf-8;
    client_max_body_size 30M;

    location / {
        try_files $uri @proxy_to_app;
    }

    location /socket.io {
      proxy_pass http://app_server;
      proxy_redirect off;

      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Upgrade websocket;
      proxy_set_header Connection "upgrade";
      proxy_read_timeout 86400;
      proxy_buffering off;
      proxy_headers_hash_max_size 1024;

    }


    location /static {
        alias /pathtowebapp/webapp/static;
    }

    location @proxy_to_app {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # enable this if and only if you use HTTPS
        proxy_set_header X-Forwarded-Proto https;

        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_set_header Host $http_host;

        # we don't want nginx trying to do something clever with
        # redirects, we set the Host: header above already.
        proxy_redirect off;
        #proxy_buffering off;
        proxy_pass http://app_server;

    }

}

I have been looking all over for examples of a websocket working with https using nginx in front of gunicorn.

My webpage loads, although the websocket connection is not successful.

The client side websocket is established using the following javascript:

var socket = io.connect('https://' + document.domain + ':' + location.port + namespace);

Here is my gunicorn.conf

import multiprocessing

bind = 'unix:/pathtowebapp/gunicorn.sock'
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = 'eventlet'

[EDIT] if I configure nginx the way it is in the Flask-IO documentation and just run (env)$ python deploy_app.py then it works. But I was under the impression that this was not as production-ideal as the setup I previously mentioned

Answer 1

The problem is that you are running multiple workers on gunicorn. This is not a configuration that is currently supported, due to the very limited load balancer in gunicorn that does not support sticky sessions. Documentation reference: https://flask-socketio.readthedocs.io/en/latest/#gunicorn-web-server.

Instead, run several gunicorn instances, each with one worker, and then set up nginx to do the load balancing, using the ip_hash method so that sessions are sticky.

Also, in case you are not aware, if you run multiple servers you need to also run a message queue, so that the processes can coordinate. This is also covered in the documentation link above.