Template toolkit character encoding

Question

it seems like template toolkit isn't handling encoding properly.

I am passing template->process a filename (where to get the template), a hash reference (containing all the parameters), and a scalar reference (where to put the output) then I am returning that and in turn displaying it to the user.

When I give it a string with umlauts in it, the html output includes a black diamond with a white question mark in place of every letter (but the correct number of letters). Any other character comes out fine.

I am using a warn to print out the string before I make the call to template->process and at this point it is fine, from what I can tell it is during the template->process call that things get turned into garbage.

Any ideas? I have tried using ENCODING => "utf8" as well as binmode => ":utf8" but neither have any affect on the output.

Here is my code with some of the fat trimmed out just to show my call to template->process, note that if I leave out the {binmode => 'utf8'} it has no effect.

<put variables in hash referenced to by vars>
<print out variables in has referenced to by $var>
my $data;
$template->process( $self->filename, $vars, \$data, {binmode => ':utf8'}) || die "Template process failed: ", $template->error();
return $data;

SOLVED Hey all thanks for your answers, the problem turned out to be that after template process had done its thing, we then wrote the string to a temporary file before outputting it so we also needed to set binmode for the file, the code now looks like:

<put variables in hash referenced to by vars>
<print out variables in has referenced to by $var>
my $data;
binmode( STDOUT, ":utf8" );
$template->process( $self->filename, $vars, \$data, {binmode => ':utf8'}) || die "Template process failed: ", $template->error();
return $data;

I thank you all for your time :)

Answer 1

The code below works. $data, specifically the strings contained must be Perl strings, i.e. properly decoded. See the introduction to encoding in the official documentation.

use Template '2.21_02';

my $tt = Template->new({
    ENCODING     => 'utf8',
    # other options …
});

$tt->process(
    $template, $data, $output, {binmode => ':utf8'}
) or die $tt->error . ' in ' . $template;

Answer 2

My solution was to punt, and all non-latin characters go in en.po to be included at runtime via [% loc('string') %] and that works fine for me since my templates had to be localized anyway.