UnicodeEncodeError when uploading files in Django admin

Question

My question is similar to the one reported here, but the proposed solution doesn't work for me. I'm trying to upload a file called 'Testaråäö.txt' via the Django admin app.

I'm running Django 1.3.1 with Gunicorn 0.13.4 and Nginx 0.7.6.7 on a Debian 6 server. Database is PostgreSQL 8.4.9. Other Unicode data is saved to the database with no problem, so I guess the problem must be with the filesystem somehow.

I've set

http {
    charset utf-8;
}

in my nginx.conf. LC_ALL and LANG is set to 'sv_SE.UTF-8'. Running 'locale' verifies this. I even tried setting LC_ALL and LANG in my nginx init script just to make sure locale is set properly.

Here's the traceback:

Traceback (most recent call last):

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/options.py", line 307, in wrapper
return self.admin_site.admin_view(view)(*args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 93, in _wrapped_view
response = view_func(request, *args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/views/decorators/cache.py", line 79, in _wrapped_view_func
response = view_func(request, *args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/sites.py", line 197, in inner
return view(request, *args, **kwargs)

File "/srv/django/letebo/app/cms/admin.py", line 81, in change_view
return super(PageAdmin, self).change_view(request, obj_id)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 28, in _wrapper
return bound_func(*args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 93, in _wrapped_view
response = view_func(request, *args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 24, in bound_func
return func(self, *args2, **kwargs2)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/transaction.py", line 217, in inner
res = func(*args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/options.py", line 985, in change_view
self.save_formset(request, form, formset, change=True)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/options.py", line 677, in save_formset
formset.save()

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/forms/models.py", line 482, in save
return self.save_existing_objects(commit) + self.save_new_objects(commit)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/forms/models.py", line 613, in save_new_objects
self.new_objects.append(self.save_new(form, commit=commit))

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/forms/models.py", line 717, in save_new
obj.save()

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/base.py", line 460, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/base.py", line 504, in save_base
self.save_base(cls=parent, origin=org, using=using)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/base.py", line 543, in save_base
for f in meta.local_fields if not isinstance(f, AutoField)]

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/fields/files.py", line 255, in pre_save
file.save(file.name, file, save=False)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/fields/files.py", line 92, in save
self.name = self.storage.save(name, content)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/files/storage.py", line 48, in save
name = self.get_available_name(name)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/files/storage.py", line 74, in get_available_name
while self.exists(name):

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/files/storage.py", line 218, in exists
return os.path.exists(self.path(name))

File "/srv/.virtualenvs/letebo/lib/python2.6/genericpath.py", line 18, in exists
st = os.stat(path)

UnicodeEncodeError: 'ascii' codec can't encode characters in position 52-54: ordinal not in range(128)

UPDATE: I tried running Gunicorn with debugging turned on, and the file uploads without any problem at all. I suppose this must mean that the issue is with Nginx. Still beats me where to look, though. Here are the raw response headers from Gunicorn and Nginx, if it makes any sense:

Gunicorn:

HTTP/1.1 302 FOUND
Server: gunicorn/0.13.4
Date: Thu, 09 Feb 2012 14:50:27 GMT
Connection: close
Transfer-Encoding: chunked
Expires: Thu, 09 Feb 2012 14:50:27 GMT
Vary: Cookie
Last-Modified: Thu, 09 Feb 2012 14:50:27 GMT
Location: http://my-server.se:8000/admin/cms/page/15/
Cache-Control: max-age=0
Content-Type: text/html; charset=utf-8
Set-Cookie: messages="yada yada yada"; Path=/

Nginx:

HTTP/1.1 500 INTERNAL SERVER ERROR
Server: nginx/0.7.67
Date: Thu, 09 Feb 2012 14:50:57 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Vary: Cookie

500

Answer 1

I had the same issue with genericpath.py giving a UnicodeEncodeError when attempting to upload a file name with non ASCII characters.

I was using nginx, uwsgi and django with python 2.7.

Everything was working OK locally but not on the server

Here are the steps I took:

1. added to /etc/nginx/nginx.conf (did not fix the problem)

   http {
        charset utf-8;
   }
   

2. I added this line to etc/default/locale (did not fix the problem)

   LANGUAGE="en_US.UTF-8"
   

3. I followed the instructions here listed under the heading 'Success' https://code.djangoproject.com/wiki/ExpectedTestFailures (did not fix the problem)

   aptitude install language-pack-en-base
   

4. Found across this ticket https://code.djangoproject.com/ticket/17816 which suggested testing a view on the server to what was happening with locale information

Django view

import locale

def locales(request):
    """Display the locales"""
    locales = "Current locale: %s %s -- Default locale: %s %s" % (
        locale.getlocale() + locale.getdefaultlocale())
    default_encoding =  sys.getdefaultencoding()
    file_system_encoding = sys.getfilesystemencoding()

    context = {
        'locales': locales,
        'default_encoding': default_encoding,
        'file_system_encoding': file_system_encoding,  # affects file uploads
    }
    return render(request, 'testing/locales.html', context)

Django Template

<h2>Locales</h2>
<p>{{ locales }}</p>

<h2>Default Encoding</h2>
<p>{{ default_encoding }}</p>

<h2>File System Encoding</h2>
<p>{{ file_system_encoding }}</p>

For me, the issue was that I had no locale and no default locale on my Ubuntu server (though I did have them on my local OSX dev machine) then files with non ASCII file names/paths will not upload correctly with python raising a UnicodeEncodeError, but only on the production server.

Solution

I added this to both my site and my site admin uwsgi config files e.g. /etc/uwsgi-emperor/vassals/my-site-config-ini file

env = LANG=en_US.utf8

Update

After moving over to docker I started getting the same error again. After looking into it further I realised in the uwsgi django instance import sys; sys.getfilesystemencoding() or {{ file_system_encoding }} above was returning ANSI_X3.4-1968, however if I started my own python instance and ran import sys; sys.getfilesystemencoding() I would get UTF-8. The ANSI_X3.4-1968 format is what is throwing the UnicodeEncodeError.

So in addition to the uwsgi solution listed in this answer above, I also had to add this to my Django dockerfile

ENV LANG en_US.UTF-8
RUN apt-get update && install -y locales && \
    sed -i -e "s/# $LANG.*/$LANG UTF-8/" /etc/locale.gen && \
    locale-gen --purge &&\
    update-locale LANG=$LANG

Or if you aren't using docker you can just run this in the shell

$ export LANG=en_US.UTF-8

Then run the RUN command above (but without RUN).

References:

http://stackoverflow.com/a/37246853/3003438
https://docs.djangoproject.com/en/dev/howto/deployment/wsgi/uwsgi/

Answer 2

It seems that it is a common problem that the service that runs the django service, be it apache, gunicorn + supervisor, daemontools, etc... doesn't always use the correct environment variables.

In this case, gunicorn is started by supervisor, so you have to explicitly tell supervisor to use utf-8 using the environment option:

environment=LANG=en_US.UTF-8, LC_ALL=en_US.UTF-8, LC_LANG=en_US.UTF-8