Django: How to ensure correct encoding of input sent via a form

Question

I'm running into an issue with Unicode input on a form with Django:

UnicodeEncodeError at /

'ascii' codec can't encode character u'\xe4' in position 7: ordinal not in range(128)

This happens with gunicorn, as well as running Django in debug mode. My form.py is unicode enabled:

#!/usr/bin/python
# -*- coding: utf-8 -*-

from django import forms
from django.core.mail import send_mail
from django.core.mail import EmailMessage
import datetime

class RTForm(forms.Form):
      # ... 
      institution_station = forms.CharField(max_length=75, label=u"Institute/Station*")
      # ...

The view is created based on that with aim to build up an e-mail from input:

class RTview(FormView):
      template_name = 'rt-form.html'
      form_class = RTForm
      success_url = '/thanks/'

      def form_valid(self, form):
          # This method is called when valid form data has been POSTed.
          # It should return an HttpResponse.
          form.send_email()
          return super(RTview, self).form_valid(form)

The function send_email looks like that:

def send_email(self):

    email = EmailMessage(
                subject='New item',
                #body='Here is the message.',
                from_email=self.cleaned_data['email'],
                to=['[email protected]'])

    # The dict fields of the form contains all defined fields incl.
    # their labels etc. So taking this we can match and check for the
    # validated data to fullfill our task.
    values = []
    for i in self.fields.iteritems():
        values.append(u"%s: \t%s" % (self[i[0]].label, str(self.cleaned_data[i[0]]) ) )

    email.body = "\n".join(values)

    email.send()

When putting unicode into institution_station, the application responds with the aforementioned error, failing at values.append() in send_email.

The traceback looks like:

Internal Server Error: /
Traceback (most recent call last):
  File "/home/frlan/.virtualenvs/gascylinders/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 114, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/frlan/.virtualenvs/gascylinders/local/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/frlan/.virtualenvs/gascylinders/local/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/frlan/.virtualenvs/gascylinders/local/lib/python2.7/site-packages/django/views/generic/edit.py", line 171, in post
    return self.form_valid(form)
  File "/home/frlan/quellen/git/gascylinders-rt-frontend/gascylinders/rtfrontend/views.py", line 12, in form_valid
    form.send_email()
  File "/home/frlan/quellen/git/gascylinders-rt-frontend/gascylinders/rtfrontend/forms.py", line 124, in send_email
    values.append(u"%s: \t%s" % (self[i[0]].label, str(self.cleaned_data[i[0]]) ) )
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 7: ordinal not in range(128)

What might I have missed?

Answer 1

I think the problem here is calling str(self.cleaned_data[i[0]]), that creates a byte string out of the unicode value. For example u"%s" % str(u"überwach!") will fail, where u"%s" % unicode(u"überwach!") will work fine (even if the unicode call is superfluous in this case...)

Edit: to add some detail: str(u"überwach!") fails with UnicodeEncodeError, because, under the hood, it calls u"überwach!".encode("ascii"), but ASCII can't handle umlauts…

Sadly, I couldn't find this beaviour described in the documentation (str() calls __str__ on a unicode object, but how is this __str__ implemented?), but for more on the topic, see the Python Unicode Howto.