how to deal with unicode in mako?

Question

I constantly get this error using mako:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 6: ordinal not in range(128)

I've told mako I'm using unicode in any possible way:

    mylookup = TemplateLookup(
        directories=['plugins/stl/templates'],
        input_encoding='utf-8',
        output_encoding='utf-8',
        default_filters=['decode.utf8'],
        encoding_errors='replace')

    self.template = Template(self.getTemplate(), lookup=mylookup,
        module_directory=tempfile.gettempdir(),
        input_encoding='utf-8',
        output_encoding='utf-8',
        default_filters=['decode.utf8'],
        encoding_errors='replace')

    html = self.template.render_unicode(data=self.stuff)

All my template files starts with:

## -*- coding: utf-8 -*-

and, inside them, all costant strings are prefixed with "u". I know the self.stuff parameter contains unicode strings, but the way I instantiate the mako objects should take care of it (otherwise what those arguments are good for?). Is there anything I forgot to do?

One more question: what's the point of encoding_errors='replace'?

=EDIT= I left only a single unicode string and this is the traceback:

Traceback (most recent call last):
  File "C:\My Dropbox\src\flucso\src\plugins\stl\main.py", line 240, in updateView
    flags=self.makoflags)
  File "C:\Python26\lib\site-packages\mako-0.3.4-py2.6.egg\mako\template.py", line 198, in render_unicode
    as_unicode=True)
  File "C:\Python26\lib\site-packages\mako-0.3.4-py2.6.egg\mako\runtime.py", line 403, in _render
    _render_context(template, callable_, context, *args, **_kwargs_for_callable(callable_, data))
  File "C:\Python26\lib\site-packages\mako-0.3.4-py2.6.egg\mako\runtime.py", line 434, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "C:\Python26\lib\site-packages\mako-0.3.4-py2.6.egg\mako\runtime.py", line 457, in _exec_template
    callable_(context, *args, **kwargs)
  File "memory:0x41317f0", line 89, in render_body
  File "C:\Python26\lib\site-packages\mako-0.3.4-py2.6.egg\mako\runtime.py", line 278, in <lambda>
    return lambda *args, **kwargs:callable_(self.context, *args, **kwargs)
  File "FriendFeed_mako", line 49, in render_inlist_entry
  File "C:\Python26\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u263c' in position 8: ordinal not in range(128)

Answer 1

finally I saved my templates in unicode, actually (I guess) utf-16 instead of utf-8. their size on disk doubled and mako started complaining about a "CompileException("Unicode decode operation of encoding 'utf-8' bla bla", so I changed the first line in all of them in:

## -*- coding: utf-16 -*-

and removed all the ".decode('utf-8')" - constant strings are still prefixed by "u".

the initializations in python now are:

mylookup = TemplateLookup(
    directories=['plugins/stl/templates'],
    input_encoding='utf-16',
    output_encoding='utf-16',
    encoding_errors='replace')

self.template = Template(self.getTemplate(), lookup=mylookup,
    module_directory=tempfile.gettempdir(),
    input_encoding='utf-16',
    output_encoding='utf-16',
    encoding_errors='replace')

it works now. looks like utf-8 was the wrong choice (or my inability to save the templates in utf-8), but I can't explain why it worked from eclipse/pydev.