Nginx and Flask-socketio Websockets: Alive but not Messaging?

Question

I've been having a bit of trouble getting Nginx to play nicely with the Python Flask-socketio library (which is based on gevent). Currently, since we're actively developing, I'm trying to get Nginx to just work as a proxy. For sending pages, I can get this to work, either by directly running the flask-socketio app, or by running through gunicorn. One hitch: the websocket messaging does not seem to work. The pages are successfully hosted and displayed. However, when I try to use the websockets, they do not work. They are alive enough that the websocket thinks it is connected, but they will not send a message. If I remove the Nginx proxy, they do work. Firefox gives me this error when I try to send a message:

Firefox can't establish a connection to the server at ws:///socket.io/1/websocket/.

Where web address is where the server is located and the unique id is just a bunch of randomish digits. It seems to be doing enough to keep the connection live (e.g., the client thinks it is connected), but can't send a message over the websocket. I have to think that the issue has to do with some part of the proxy, but am having mighty trouble debugging what the issue might be (in part because this is my first go-round with both Flask-socketIO and nginx). The configuration file I am using for nginx is:

user       <user name>;  ## This is set to the user name for the remote SSH session
worker_processes  5;

events {
  worker_connections  1024;  ## Default: 1024
}

http {
  default_type application/octet-stream;
  log_format   main '$remote_addr - $remote_user [$time_local]  $status '
    '"$request" $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';
  sendfile     on;
  server_names_hash_bucket_size 128; # this seems to be required for some vhosts

  server {
    listen 80;
    server_name _;
    location / {
        proxy_pass http://localhost:8000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
    }
  } 
}

I made the config file as an amalgam of a general example and a websocket specific one, but trying to fiddle with it has not solved the issue. Also, I am using the werkzeug Proxy_Fix call on my Flask app.wsgi_app when I use it in wsgi mode. I've tried it with and without that, to no avail, however. If anyone has some insight, I will be all ears/eyes.

Answer 1

I managed to fix this. The issues were not specific to flask-socketio, but they were specific to Ubuntu, NginX, and gevent-socketio. Two significant issues were present:

1. Ubuntu 12.04 has a truly ancient version of nginx (1.1.19 vs 1.6.x for stable versions). Why? Who knows. What we do know is that this version does not support websockets in any useful way, as 1.3.13 is about the earliest you should be using.
2. By default, gevent-socketio expects your sockets to be at the location /socket.io . You can upgrade the whole HTTP connection, but I had some trouble getting that to work properly (especially after I threw SSL into the mix).
3. I fixed #1, but in fiddling with it I purged by nginx and apt-get installed... the default version of nginx on Ubuntu. Then, I was mysteriously confused as to why things worked even worse than before. Many .conf files valiantly lost their lives in this battle.

If trying to debug websockets in this configuration, I would recommend the following steps:

1. Check your nginx version via 'nginx -v'. If it is anything less than 1.4, upgrade it.
2. Check your nginx.conf settings. You need to make sure the connection upgrades.
3. Check that your server IP and port match your nginx.conf reverse proxy.
4. Check that your client (e.g., socketio.js) connects to the right location and port, with the right protocol.
5. Check your blocked ports. I was on EC2, so you have to manually open 80 (HTTP) and 443 (SSL/HTTPS).

Having just checked all of these things, there are takeaways.

1. Upgrading to the latest stable nginx version on Ubuntu (full ref) can be done by:

   sudo apt-get install python-software-properties
   sudo apt-get install software-properties-common
   sudo add-apt-repository ppa:nginx/stable
   sudo apt-get update
   sudo apt-get install nginx
   

   In systems like Windows, you can use an installer and will be less likely to get a bad version.

2. Many config files for this can be confusing, since nginx officially added sockets in about 2013, making earlier workaround configs obsolete. Existing config files don't tend to cover all the bases for nginx, gevent-socketio, and SSL together, but have them all separately (Nginx Tutorial, Gevent-socketio, Node.js with SSL). A config file for nginx 1.6 with flask-socketio (which wraps gevent-socketio) and SSL is:

   user <user account, probably optional>;
   worker_processes  2;
   error_log  /var/log/nginx/error.log;
   pid        /var/run/nginx.pid;
   
   events {
       worker_connections  1024;
   }
   
   http {
       include mime.types;
       default_type       application/octet-stream;
       access_log         /var/log/nginx/access.log;
       sendfile           on;
   #   tcp_nopush         on;
       keepalive_timeout  3;
   #   tcp_nodelay        on;
   #   gzip               on;
       client_max_body_size 20m;
       index              index.html;
   
       map $http_upgrade $connection_upgrade {
               default upgrade;
               ''      close;
       }
   
       server {
         # Listen on 80 and 443
         listen 80 default;
         listen 443 ssl;  (only needed if you want SSL/HTTPS)
         server_name <your server name here, optional unless you use SSL>;
   
         # SSL Certificate (only needed if you want SSL/HTTPS)
         ssl_certificate <file location for your unified .crt file>;
         ssl_certificate_key <file location for your .key file>;
   
         # Optional: Redirect all non-SSL traffic to SSL. (if you want ONLY SSL/HTTPS)
         # if ($ssl_protocol = "") {
         #   rewrite ^ https://$host$request_uri? permanent;
         # }
   
         # Split off basic traffic to backends
         location / {
           proxy_pass http://localhost:8081; # 127.0.0.1 is preferred, actually.
           proxy_redirect off;
         }
   
         location /socket.io {
           proxy_pass          http://127.0.0.1:8081/socket.io; # 127.0.0.1 is preferred, actually.
           proxy_redirect off;
           proxy_buffering off; # Optional
           proxy_http_version 1.1;
           proxy_set_header Upgrade $http_upgrade;
           proxy_set_header Connection $connection_upgrade;
         }
       }
    }
   

3. Checking that your Flask-socketio is using the right port is easy. This is sufficient to work with the above:

   from flask import Flask, render_template, session, request, abort
   import flask.ext.socketio
   FLASK_CORE_APP = Flask(__name__)
   FLASK_CORE_APP.config['SECRET_KEY'] = '12345' # Luggage combination
   SOCKET_IO_CORE = flask.ext.socketio.SocketIO(FLASK_CORE_APP)
   
   @FLASK_CORE_APP.route('/')
   def index():
       return render_template('index.html')
   
   @SOCKET_IO_CORE.on('message')
   def receive_message(message):
       return "Echo: %s"%(message,)
   
   SOCKET_IO_CORE.run(FLASK_CORE_APP, host=127.0.0.1, port=8081)
   

4. For a client such as socketio.js, connecting should be easy. For example:

   <script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/socket.io/0.9.16/socket.io.min.js"></script>
   <script type="text/javascript">
       var url = window.location.protocol + document.domain + ':' + location.port,
           socket = io.connect(url);
       socket.on('message', alert);
       io.emit("message", "Test")
   </script>
   

5. Opening ports is really more of a server-fault or a superuser issue, since it will depend a lot on your firewall. For Amazon EC2, see here.

6. If trying all of this does not work, cry. Then return to the top of the list. Because you might just have accidentally reinstalled an older version of nginx.