UnicodeEncodeError when saving ImageField containing non-ASCII characters in Django admin

Question

I'm trying to upload an image file in django admin inlines and getting UnicodeEncodeError when trying to upload a file with a filename containing non-ascii characters:

 File "/usr/local/lib/python2.6/site-packages/django/db/models/fields/files.py", line 92, in save
   self.name = self.storage.save(name, content)

 File "/usr/local/lib/python2.6/site-packages/django/core/files/storage.py", line 47, in save
   name = self.get_available_name(name)

 File "/usr/local/lib/python2.6/site-packages/django/core/files/storage.py", line 73, in get_available_name
   while self.exists(name):

 File "/usr/local/lib/python2.6/site-packages/django/core/files/storage.py", line 196, in exists
   return os.path.exists(self.path(name))

 File "/usr/local/lib/python2.6/genericpath.py", line 18, in exists
   st = os.stat(path)

There is a paragraph about this issue in Django docs: http://docs.djangoproject.com/en/dev/howto/deployment/modpython/#if-you-get-a-unicodeencodeerror - they say I must define LANG and LC_ALL env variables, plus defining them using os.env won't work. So I've defined them in my .htaccess file and I'm sure they are there:

META
Variable    Value
CONTENT_LENGTH  '27289'
...
LANG    'en_US.UTF-8'
LC_ALL  'en_US.UTF-8'
LC_LANG     'en_US.UTF-8'

The problem still exists. Django version is 1.2.3 (latest stable), sys.getfilesystemencoding() (which I believe is relevant to the issue) returns "ANSI_X3.4-1968".

The model/admin code is nothing special: an ArticleImage model with ImageField, and ArticleAdmin containing ArticleImage inlines.

UPDATE I couldn't fix this issue so I've given up using apache setup and started the application using runfcgi + nginx. Uploads work fine now but I'm not adding this as a solution because the question was about apache.

Answer 1

On Debian (Lenny) you simply add the following two lines to /etc/apache2/envvars:

export LANG='en_GB.UTF-8'
export LC_ALL='en_GB.UTF-8'

...that's for UK web servers. For US:

export LANG='en_US.UTF-8'
export LC_ALL='en_US.UTF-8'

And restart Apache.

Answer 2

You should try defining the LANG and LC_ALL for the whole Apache 2 environment.

For my deployments I also make sure that the python default system encoding is set to utf-8 as well.

For the Python default encoding I usually create/edit sitecustomize.py, see http://blog.ianbicking.org/illusive-setdefaultencoding.html

As for Apache - there is line in init script /etc/init.d/apache2 (Ubuntu 8.04 LTS) that creates the environment. I added the correct LC_ALL, LANG there. Basically it should be in the server init scripts somewhere for all the OSes.