Perl - File Encoding and Word Comparison

Question

I have a file with one phrase/terms each line which i read to perl from STDIN. I have a list of stopwords (like "á", "são", "é") and i want to compare each one of them with each term, and remove if they are equal. The problem is that i'm not certain of the file's encoding format.

I get this from the file command:

words.txt: Non-ISO extended-ASCII English text

My linux terminal is in UTF-8 and it shows the right content for some words and for others don't. Here is the output from some of them:

condi<E3>
conte<FA>dos
ajuda, mas não resolve
mo<E7>ambique
pedagógico são fenómenos

You can see that the 3rd and 5th lines are correctly identifying words with accents and special characters while others don't. The correct output for the other lines should be: condiã, conteúdos and moçambique.

If i use binmode(STDOUT, utf8) the "incorrect" lines now output correctly while the other ones don't. For example the 3rd line:

ajuda, mas não resolve

What should i do guys?

Answer 1

I strongly suggest you create a filter that takes a file with lines in mixed encodings and translates them to pure UTF-8. Then instead

open(INPUT, "< badstuff.txt") || die "open failed: $!";

you would open either the fixed version, or a pipe from the fixer, like:

open(INPUT, "fixit < badstuff.txt |") || die "open failed: $!"

In either event, you would then

binmode(INPUT, ":encoding(UTF-8)") || die "binmode failed";

Then the fixit program could just do this:

use strict;
use warnings;
use Encode qw(decode FB_CROAK);

binmode(STDIN,  ":raw")  || die "can't binmode STDIN";
binmode(STDOUT, ":utf8") || die "can't binmode STDOUT";

while (my $line = <STDIN>) {
    $line = eval { decode("UTF-8", $line, FB_CROAK() };
    if ($@) { 
        $line = decode("CP1252", $line, FB_CROAK()); # no eval{}!
    }
    $line =~ s/\R\z/\n/;  # fix raw mode reads
    print STDOUT $line;    
}

close(STDIN)  || die "can't close STDIN: $!";
close(STDOUT) || die "can't close STDOUT: $!";
exit 0;

See how that works? Of course, you could change it to default to some other encoding, or have multiple fall backs. Probably it would be best to take a list of them in @ARGV.

Answer 2

It works like this:

C:\Dev\Perl :: chcp
Aktive Codepage: 1252.

C:\Dev\Perl :: type mixed-encoding.txt
eins zwei drei Käse vier fünf Wurst
eins zwei drei Käse vier fünf Wurst

C:\Dev\Perl :: perl mixed-encoding.pl < mixed-encoding.txt
eins zwei drei vier fünf
eins zwei drei vier fünf

Where mixed-encoding.pl goes like this:

use strict;
use warnings;
use utf8; # source in UTF-8
use Encode 'decode_utf8';
use List::MoreUtils 'any';

my @stopwords = qw( Käse Wurst );

while ( <> ) { # read octets
    chomp;
    my @tokens;
    for ( split /\s+/ ) {
        # Try UTF-8 first. If that fails, assume legacy Latin-1.
        my $token = eval { decode_utf8 $_, Encode::FB_CROAK };
        $token = $_ if $@;
        push @tokens, $token unless any { $token eq $_ } @stopwords;
    }
    print "@tokens\n";
}

Note that the script doesn't have to be encoded in UTF-8. It's just that if you have funky character data in your script you have to make sure the encoding matches, so use utf8 if your encoding is UTF-8, and don't if it isn't.

Update based on tchrist's sound advice:

use strict;
use warnings;
# source in Latin1
use Encode 'decode';
use List::MoreUtils 'any';

my @stopwords = qw( Käse Wurst );

while ( <> ) { # read octets
        chomp;
        my @tokens;
        for ( split /\s+/ ) {
                # Try UTF-8 first. If that fails, assume 8-bit encoding.
                my $token = eval { decode utf8 => $_, Encode::FB_CROAK };
                $token    = decode Windows1252 => $_, Encode::FB_CROAK if $@;
                push @tokens, uc $token unless any { $token eq $_ } @stopwords;
        }
        print "@tokens\n";
}