Why doesn't "\w" match Unicode word characters (for example, "ğ,İ,ş,ç,ö,ü") in Perl regular expression?

Question

Why doesn't "\w" match Unicode word characters (for example, "ğ,İ,ş,ç,ö,ü") in a Perl regular expression?

I tried to include these characters in regular expression m{\w+}g. However, it does not match "ğ,İ,ş,ç,ö,ü".

How can I make this work?

use strict;
use warnings;
use v5.12;
use utf8;

open(MYINPUTFILE, "< $ARGV[0]");

my @strings;
my $delimiter;
my $extensions;
my $id;

while(<MYINPUTFILE>)
{
    my($line) = $_;
    chomp($line);
    print $line."\n";
    unshift(@strings,$line =~ /\w+/g);
    $delimiter = /[._\s]/;
    $extensions = /pdf$|doc$|docx$/;
    $id = /^200|^201/;
}

foreach(@strings){
    print $_."\n";
}

The input file is like:

Çidem_Şener
Hüsnü Tağlip
...

The output goes like:

H�

sn�

Ta�

lip

�

idem_�

ener

In the code, I try to read the file and take each string in the array. (Delimiter can be _ or . or \s).

Answer 1

Make sure that Perl is treating the data as UTF-8.

e.g. if it is embedded in the script itself:

#!/usr/bin/perl

use strict;
use warnings; 
use v5.12;
use utf8;   # States that the Perl program itself is saved using utf8 encoding

say "matched" if "ğİşçöü" =~ /^\w+$/;

That outputs matched. If I remove the use utf8; line, it does not.

Answer 2

\w matches any of ğ İ ş ç ö ü just fine.

'ğİşçöü' =~ /\A \w+ \z/msx;     # true

You probably made a mistake and forgot to decode input from octets into Perl characters. I suspect your regex examines stuff on the byte level instead of the character level, like one would expect.

Read http://p3rl.org/UNI and http://training.perl.com/scripts/perlunicook.html to learn about the topic of encoding in Perl.

---

Edit:

The problem is likely here (I cannot tell for sure without the content of the file):

open(MYINPUTFILE, "< $ARGV[0]");

Find out the encoding of the file, perhaps it's UTF-8 or Windows-1254. Rewrite it, e.g.:

open $in, '<:utf8', $ARGV[0];
open $in, '<:encoding(Windows-1254)', $ARGV[0];

Similarly, printing characters out to STDOUT (near the end of your program) is similarly broken because of the lack of encoding. ℞ 16: Declare STD{IN,OUT,ERR} to be in locale encoding shows one way how to do it properly.