uWSGI works as process but not as daemon

Question

For my current flask deployment, I had to set up a uwsgi server. This is how I have created the uwsgi daemon:

sudo vim /etc/init/uwsgi.conf

# file: /etc/init/uwsgi.conf
description "uWSGI server"

start on runlevel [2345]
stop on runlevel [!2345]
respawn

exec /myproject/myproject-env/bin/uwsgi --uid www-data --gid www-data --home /myproject/myproject-env/site/F11/Engineering/  --socket /tmp/uwsgi.sock --chmod-socket --module F11 --callable app --pythonpath /myproject/myproject-env/site/F11/Engineering/ -H /myproject/myproject-env

However after running this successfully: sudo start uwsgi

uwsgi start/running, process 1286

And trying to access the application via browser:

I get a 502 Bad Gateway

and an error entry in nginx error.log:

2013/06/13 23:47:28 [error] 743#0: *296 upstream prematurely closed connection while reading response header from upstream, client: xx.161.xx.228, server: myproject.com, request: "GET /show_records/2013/6 HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "myproject.com"

But the sock file has the permission it needs:

srw-rw-rw- 1 www-data www-data 0 Jun 13 23:46 /tmp/uwsgi.sock

If I run the exec command from above in the command line as a process, it works perfectly fine. Why is the daemon not working correctly please?

btw Nginx is running as vim /etc/nginx/nginx.conf

user www-data;

and vim /etc/nginx/sites-available/default

location / {
                uwsgi_pass   unix:///tmp/uwsgi.sock;
                include        uwsgi_params;
        }

and it is started as sudo service nginx start

I am running this on Ubuntu 12.04 LTS.

I hope I have provided all the necessary data, hope someone can guide me into the right direction. Thanks.

Answer 1

Finally I have solved this problem after working nearly 2 days on it. I hope this solution will help other flask/uwsgi users that are experiencing a similar problem.

I had two major issues that caused this.

1) The best way to find the problems with a daemon is obviously a log file and a cleaner structure.

sudo vim /etc/init/uwsgi.conf

Change the daemon script to the following:

# file: /etc/init/uwsgi.conf
description "uWSGI server"

start on runlevel [2345]
stop on runlevel [!2345]
respawn
exec /home/ubuntu/uwsgi-1.9.12/uwsgi -c /myproject/uwsgi.ini

vim /myproject/uwsgi.ini

[uwsgi]
socket = /tmp/uwsgi.sock
master = true
enable-threads = true
processes = 5
chdir= /myproject/F11/Engineering
module=F11:app
virtualenv = /myproject/myproject-env/
uid =  www-data
gid = www-data
logto = /myproject/error.log

This is much cleaner way of setting up the daemon. Also notice the last line how to setup the log file. Initially I had set the log file to /var/log/uwsgi/error.log. After a lot of sweat and tears I realized the daemon is running as www-data and hence can not access the /var/log/uwsgi/error.log since the error.log was owned by root:root. This made the uwsgi fail silently.

I found it much more efficient to just point the log file to my own /myproject, where the daemon has guaranteed access as www-data. And also don't forget to make the whole project accessible to www-data or the daemon will fail with an Internal Server error message. -->

sudo chown www-data:www-data -R /myproject/

Restart uwsgi daemon:

sudo service uwsgi restart

2) Now you have three log files to lookout for:

• tail -f /var/log/upstart/uwsgi.log --> Shows problems with your daemon upon start

• tail -f /var/log/nginx/error.log --> Shows permission problems when wsgi access is refused, often because /tmp/uwsgi.sock file is owned by root instead of www-data. In that case simply delete the sock file sudo rm /tmp/uwsgi.sock

• tail -f /myproject/error.log --> Shows errors thrown by uwsgi in your application

This combination of log files helped me to figure out that I also had a bad import with Flask-Babel in my Flask application. Bad in that sense, that the way I utilized the library was falling back to the system's locale to determine the datetime format.

File "/myproject/F11/Engineering/f11_app/templates/show_records.html", line 25, in block "body"
    <td>{{ record.record_date|format_date }}</td>
  File "./f11_app/filters.py", line 7, in format_date
    day = babel_dates.format_date(value, "EE")
  File "/myproject/myproject-env/local/lib/python2.7/site-packages/babel/dates.py", line 459, in format_date
    return pattern.apply(date, locale)
  File "/myproject/myproject-env/local/lib/python2.7/site-packages/babel/dates.py", line 702, in apply
    return self % DateTimeFormat(datetime, locale)
  File "/myproject/myproject-env/local/lib/python2.7/site-packages/babel/dates.py", line 699, in __mod__
    return self.format % other
  File "/myproject/myproject-env/local/lib/python2.7/site-packages/babel/dates.py", line 734, in __getitem__
    return self.format_weekday(char, num)
  File "/myproject/myproject-env/local/lib/python2.7/site-packages/babel/dates.py", line 821, in format_weekday
    return get_day_names(width, context, self.locale)[weekday]
  File "/myproject/myproject-env/local/lib/python2.7/site-packages/babel/dates.py", line 69, in get_day_names
    return Locale.parse(locale).days[context][width]
AttributeError: 'NoneType' object has no attribute 'days'

This is the way I was using the Flask filter:

import babel.dates as babel_dates

@app.template_filter('format_date')
def format_date(value):
    day = babel_dates.format_date(value, "EE")
    return '{0} {1}'.format(day.upper(), affix(value.day))

The strangest part is that this code is working perfectly fine within the dev environment (!). It works even fine when running the uwsgi as a root process from the command line. But it fails when ran by the www-data daemon. This must have something to do with how the locale is set, which Flask-Babel is trying to fall back to.

When I changed the import like this, it all worked finally with the daemon:

from flask.ext.babel import format_date  

@app.template_filter('format_date1')
def format_date1(value):
    day = format_date(value, "EE")
    return '{0} {1}'.format(day.upper(), affix(value.day))

Hence be careful when using Eclipse/Aptana Studio that is trying to pick the right namespace for your classes in code. It can really turn ugly.

It is now working perfectly fine as a uwsgi daemon on an Amazon Ec2 (Ubuntu 12.04) since 2 days. I hope this experience helps fellow python developers.