Use of 'use utf8;' gives me 'Wide character in print'

Question

If I run the following Perl program:

perl -e 'use utf8; print "鸡\n";' 

I get this warning:

Wide character in print at -e line 1. 

If I run this Perl program:

perl -e 'print "鸡\n";' 

I do not get a warning.

I thought use utf8 was required to use UTF-8 characters in a Perl script. Why does this not work and how can I fix it? I'm using Perl 5.16.2. I have the same issue if this is in a file instead of being a one liner on the command line.

Answer 1

Without use utf8 Perl interprets your string as a sequence of single byte characters. There are four bytes in your string as you can see from this:

$ perl -E 'say join ":", map { ord } split //, "鸡\n";' 233:184:161:10 

The first three bytes make up your character, the last one is the line-feed.

The call to print sends these four characters to STDOUT. Your console then works out how to display these characters. If your console is set to use UTF8, then it will interpret those three bytes as your single character and that is what is displayed.

If we add in the utf8 module, things are different. In this case, Perl interprets your string as just two characters.

$ perl -Mutf8 -E 'say join ":", map { ord } split //, "鸡\n";' 40481:10 

By default, Perl's IO layer assumes that it is working with single-byte characters. So when you try to print a multi-byte character, Perl thinks that something is wrong and gives you a warning. As ever, you can get more explanation for this error by including use diagnostics. It will say this:

(S utf8) Perl met a wide character (>255) when it wasn't expecting one. This warning is by default on for I/O (like print). The easiest way to quiet this warning is simply to add the :utf8 layer to the output, e.g. binmode STDOUT, ':utf8'. Another way to turn off the warning is to add no warnings 'utf8'; but that is often closer to cheating. In general, you are supposed to explicitly mark the filehandle with an encoding, see open and perlfunc/binmode.

As others have pointed out you need to tell Perl to accept multi-byte output. There are many ways to do this (see the Perl Unicode Tutorial for some examples). One of the simplest ways is to use the -CS command line flag - which tells the three standard filehandles (STDIN, STDOUT and STDERR) to deal with UTF8.

$ perl -Mutf8 -e 'print "鸡\n";' Wide character in print at -e line 1. 鸡 

vs

$ perl -Mutf8 -CS -e 'print "鸡\n";' 鸡 

Unicode is a big and complex area. As you've seen, many simple programs appear to do the right thing, but for the wrong reasons. When you start to fix part of the program, things will often get worse until you've fixed all of the program.