UnicodeError error when calling Django i18n makemessages command

Question

I'm working with Django's internationalisation features to generate translation strings for a webapp.

A problem is arising where I try and call makemessages, and the existing language .po file contains a special character (such as $, £, etc).

Where one of these exists, makemessages tries to load the existing .po file and to decode it. When it does this, I get an error:

Traceback (most recent call last):
 File "manage.py", line 18, in <module>
   execute_from_command_line(sys.argv)
 File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
   utility.execute()
 File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 346, in execute
   self.fetch_command(subcommand).run_from_argv(self.argv)
 File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 394, in run_from_argv
   self.execute(*args, **cmd_options)
 File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 445, in execute
   output = self.handle(*args, **options)
 File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/makemessages.py", line 325, in handle
   self.write_po_file(potfile, locale)
 File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/makemessages.py", line 458, in write_po_file
   msgs, errors, status = gettext_popen_wrapper(args)
 File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/makemessages.py", line 51, in gettext_popen_wrapper
   stdout = stdout.decode(stdout_encoding)
 File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
   return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa2' in position 2105: ordinal not in range(128)

I've tried to dig back through the traceback here, but I'm at a loss as to what's happening.

It seems as though Django tries to decode the existing .po file as UTF8, but then when re-encoding it, it's using an ASCII codec.

Any insights as to what's wrong would be massively appreciated.

---

Edit:

• OS: Ubuntu 15.10 and OS X 10.11.6
• Python: 2.7.10 and 2.7.11
• Django: 1.8.14
• Six: 1.10.0

I've tried reinstalling Django/Six as suggested, but the error is still there.

Ubuntu's localedef --list-archive:

en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8

Content-type of the problematic translation file:

 "Content-Type: text/plain; charset=UTF-8\n"

Answer 1

Note this is a different exception location from this similar question mentioned in the comments.

It seems to me the only way this can happen is if there's been a modification to your django install or there's a bug in the python 2.7 version.

Your stack is:

> msgs, errors, status = gettext_popen_wrapper(args)
> stdout = stdout.decode(stdout_encoding)

gettext_popen_wrapper (on django 1.8, which is what I think you're using, can you confirm?) and popen_wrapper which creates stdout (after removing comments/docstrings and reindenting for clarity, see popen_wrapper and gettext_popen_wrapper on github for the unadulterated code):

def popen_wrapper(args, os_err_exc_type=CommandError, universal_newlines=True):
    try:
        p = Popen(args, shell=False, stdout=PIPE, stderr=PIPE,
                close_fds=os.name != 'nt', universal_newlines=universal_newlines)
    except OSError as e:
        strerror = force_text(e.strerror, DEFAULT_LOCALE_ENCODING,
                              strings_only=True)
        six.reraise(os_err_exc_type, os_err_exc_type('Error executing %s: %s' %
                    (args[0], strerror)), sys.exc_info()[2])
    # NB: subprocess.Popen.communicate() should return two bytes 
    # (i.e. str in python 2) objects
    output, errors = p.communicate()
    return (
        output,
        force_text(errors, DEFAULT_LOCALE_ENCODING, strings_only=True),
        p.returncode
    )

def gettext_popen_wrapper(args, 
                          os_err_exc_type=CommandError, 
                          stdout_encoding="utf-8"):
    manual_io_wrapper = six.PY3 and stdout_encoding != DEFAULT_LOCALE_ENCODING

    stdout, stderr, status_code = popen_wrapper(
        args, os_err_exc_type=os_err_exc_type,
        universal_newlines=not manual_io_wrapper)

    if manual_io_wrapper:
        stdout = io.TextIOWrapper(io.BytesIO(stdout), encoding=stdout_encoding).read()
    if six.PY2:
        # EXCEPTION HIT ON THE FOLLOWING LINE
        stdout = stdout.decode(stdout_encoding)
    return stdout, stderr, status_code

So stdout should be a plain str object (i.e. a bunch of bytes needing decoding) by the time we call stdout.decode(). However, if that were the case then why is the exception in encoding? We would only need to encode if the object already was a unicode object, i.e. if it was of type unicode. And sure enough, if we add the line

stdout = stdout.decode('utf-8')

before

stdout = stdout.decode(stdout_encoding)

Then now the decode method first attempts to encode the unicode stdout, using the default encoding of ascii, which causes the exception you've seen. I also got the same error by setting manual_io_wrapper to True, which caused the stdout = io.TextWrapper(...) line to happen as well (which produces a unicode as well), but that shouldn't be True because you're on python 2 not 3.

So I think either:

• You've got a bad install of django or six, or it's been edited. Try reinstalling them.
• You've hit a bug in subprocess.Popen.communicate() and for some reason it's returning a unicode not a str (I believe in python 3 that is possible if universal_newlines are turned on. You may get mileage by reinstalling python or upgrading to a later version.

My main point though is that I don't think it's an environment issue. It would be interesting to know for any follow-ups:

• what platform you're on
• what python 2.7 you're using
• what django you're using.