Nginx + Gunicorn + Flask --> constant 500 errors after X hours of usage

Question

I have a Flask app that is running with Nginx + Gunicorn normally without problems. The app is still in development so there is only about one request per hour at the moment.

The problem is that Gunicorn seems to suddenly crash always after about 12-36 hours from the last restart. When this happens, nginx can still serve static files but anything that needs Gunicorn starts to return 500 errors even though a Gunicorn process is still running on the server. The problem is always fixed for next 12-36 hours by restarting Gunicorn by sudo supervisorctl restart xxx (no nginx restart needed). The problem has occurred about 10 times so far. Would it be possible to improve logging somehow or do something else?

Nginx conf (/etc/nginx/sites-available/xxx_gunicorn):

server {
        listen 80 default_server;
        listen [::]:80 default_server ipv6only=on;

        access_log /var/log/nginx/xxx-access.log;

        server_name 127.0.0.1 www.xxx.yy;

        location / {
                proxy_pass_header Server;
                proxy_set_header Host $http_host;
                proxy_redirect off;
                proxy_set_header X-Forwarded-For  $remote_addr;
                proxy_set_header X-Scheme $scheme;
                proxy_connect_timeout 10;
                proxy_read_timeout 10;
                proxy_pass http://127.0.0.1:8000/;
        }

        location /static {
                alias  /opt/xxx/static/;
        }
}

supervisor conf to start gunicorn (/etc/supervisor/conf.d/xxx.conf)

[program:xxx]
command = gunicorn xxx:app -b localhost:8000 --debug --log-level debug --log-file /var/log/gunicorn.log --error-logfile /var/log/gunicorn.error.log --workers 2 --worker-connections 1000 --max-requests 100 --timeout 30
directory = /opt/xxx
user = ubuntu
stdout_logfile = /var/log/gunicorn.log ; Where to write log messages
autostart=true
autorestart=true
redirect_stderr=true

xxx.py

...
app = Flask(__name__)
...
if __name__ == '__main__':
    app.debug=True
    app.run()

/var/log/gunicorn.error.log : Nothing is logged when the 500 errors begin. No suspicious lines before the 500 errors either.

/var/log/gunicorn.log : No suspicious lines.

/var/log/nginx/error.log : No suspicious lines.

/var/log/nginx/xxx-access.log : the 500 errors are visible here:

80.221.255.134 - - [27/Jan/2015:07:01:50 +0000] "GET / HTTP/1.1" 500 291 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36"

Answer 1

As I know gunicorn haven't option log-file, only access-logfile and error-logfile:

http://gunicorn-docs.readthedocs.org/en/latest/settings.html#logging

But to log errors in your flask application you have to setup python logger in it. For example, it can be done like this:

from logging.handlers import WatchedFileHandler

@app.before_first_request
def setup_logging():
    """
    Setup logging
    """
    handler = WatchedFileHandler("/var/log/your_flask_app.log")
    app.logger.addHandler(handler)
    app.logger.setLevel(logging.INFO)