"utf8 "\x96" does not map to Unicode at <somefile.pl> at line no - 321" Error in Perl

Question

I am setting up Perl application . I am getting this error. "utf8 "\x96" does not map to Unicode at" Can anybody let me know the cause and solution. Am i missing any configuration or it is my installation problem ?

Following is the code :

open(FILE,"<:encoding(UTF-8)",$self->{BASEDIR}.$self->{FILENAME}) || die "could not open file $basedir$filename - $!";

Answer 1

The character 0x96 is not a valid UTF-8 encoding. There is a block of code points just above 0x80 that, in UTF-8, encodes the start of a 2- or 3-byte character.

The input you are reading must not be UTF-8, and is most likely Latin1 or CP1252.

You will need to convert the input data to UTF-8, however one does that in Perl (it's been a long time since I did any Perl and it didn't use UTF-8 by default when I was writing Perl :-)

Answer 2

I suspect that something you believe to be encoded in UTF-8 is not, in fact, encoded with UTF-8.

Answer 3

Just putting this info out there in case it helps someone in the future.

If you're working with a Microsoft product, this can be caused by non-US characters (European, Chinese, etc). For instance, if someone sends you an excel spreadsheet of data that you need to process and it's saved in .csv format, those characters can be outside of the utf-8 range if it wasn't saved properly.

Fortunately, at least in Excel for Mac v. 15, it is possible to take that data and "save as" specifically a CSV UTF-8 file - it's in the list of options. This is a separate option from the other CSV file option. This will convert non-US characters into the UTF-8 charset and solve this issue.