WWW::Mechanize and wide character warning

Question

When I trying to download some HTML file with the code below:

$mech->get($link)
$mech->save_content("file.html");

I get the warning:

Wide character in print at C:/strawberry/perl/site/lib/WWW/Mechanize.pm line 2040.

Could someone explain how I can repair this warning?

Answer 1

You'll need to ensure that the output filehandles are opened with the proper encoding.

From a brief glance at the docs, it doesn't look like Mech has configurable encodings for saved files, so you can grab the content and save it yourself:

$mech->get( $link );
my $content = $mech->content;

open my $fh, '>:utf8', $file or die "$file: $!";
print $fh $content;

The :utf8 bit in the open will make sure that data sent to the filehandle is encoded properly as UTF-8.

Another way to do it is to encode manually:

use Encode;
my $content = encode 'utf8', $mech->content;

open my $fh, '>', $file or die "$file: $!";
binmode $fh;
print $fh $content;

Answer 2

Prior to version 1.73, you had to save the content manually using the solution posted by @friedo.

Since then, save_content() lets you set the I/O layer that is used by Mechanize when opening the filehandle. By setting binmode to :utf8 as follows, wide characters are written without warning:

$mech->save_content("file.html", binmode => ':utf8');