How to get unicode code points in perl v5.24?

Question

I want to document the hex unicode code points for strings that are cut and pasted into bash as an argument. ord does not do this; ord seems to only work within ascii bounds.

Most of what I've found regarding ord, is at least six years old, or older, and no longer relevant, as I am using v5.24 which I've read has unicode support build in. In python it is trivial:

for i in unicode(sys.argv[1], 'utf-8'):
    print i.encode("utf_16_be").encode("hex")

which works from bash. I think the problem is with the ord function itself, which does not seem updated for unicode.

# ord.pl does not provide the unicode code point for a pasted variable.
use strict;
use warnings;
#use charnames (); #nope.
#use feature 'unicode_strings'; #nope.  Already automatically using as of v5.12.
#use utf8; #nope.
#binmode(STDOUT, ":encoding(UTF-8)"); #nope.

my $arg = "";

foreach $arg  (@ARGV) {
  print $arg . " is " . ord($arg) . " in code.\n";  # seems to me ord is ascii only.
  #utf8::encode($arg);  #nope.
  #print unpack("H*", $arg) . "\n";  #nope.

  #printf "%vX\n", $arg;  #nope.
}

which gets:

david@A8DT01:~/bin$ ord.pl A B C D a b c d \  \\ … —  €
A is 65 in code.
41
B is 66 in code.
42
C is 67 in code.
43
D is 68 in code.
44
a is 97 in code.
61
b is 98 in code.
62
c is 99 in code.
63
d is 100 in code.
64
  is 32 in code.
20
\ is 92 in code.
5c
… is 226 in code.
c3a2c280c2a6
— is 226 in code.
c3a2c280c294
 is 239 in code.
c3afc280c2a8
€ is 226 in code.
c3a2c282c2ac
david@A8DT01:~/bin$

I'd like to get the output I get in python:

david@A8DT01:~/bin$ python code-points.py "ABCDabcd \ … —  €"
0041
0042
0043
0044
0061
0062
0063
0064
0020
005c
0020
2026
0020
2014
0020
f028
0020
20ac
david@A8DT01:~/bin$

Answer 1

It is not a problem with ord, but with encoding. Input from the commandline will usually be UTF-8 encoded, and ord only takes a single character, not a multi byte string. You can use the -CA switch to decode @ARGV automatically (or -CSA so that STDOUT is also encoded for the terminal), or do it in the script.

use strict;
use warnings;
use Encode;
foreach my $arg (@ARGV) {
  my $decoded = decode 'UTF-8', $arg;
  print $arg . " is " . ord($decoded) . " in code.\n";
}

However, your python script is doing something very different, it is returning the hex representation of the string encoded to UTF-16BE, not the decimal ordinals of the unicode characters. You can do this also in Perl.

use strict;
use warnings;
use Encode;
foreach my $arg (@ARGV) {
  my $utf16 = encode 'UTF-16BE', decode 'UTF-8', $arg;
  print $arg . " is " . sprintf("%vX", $utf16) . " in code.\n";
}

Answer 2

The Perl equivalent of

for ucp_str in unicode(sys.argv[1], 'utf-8'):
    print ucp_str.encode("utf_16_be").encode("hex")

is

use Encode qw( decode encode );

for my $ucp_str (split(//, decode("UTF-8", $ARGV[0]))) {
   say unpack("H*", encode("UTF-16be", $ucp_str));
}

Demo:

$ ./a.py aé€♠𠀀
0061
00e9
20ac
2660
d840dc00

$ ./a.pl aé€♠𠀀
0061
00e9
20ac
2660
d840dc00

But you asked to output the code points, and that's not what those programs do. For that, you can use the following:

use Encode qw( decode_utf8 );

for my $ucp_num (unpack('W*', decode_utf8($ARGV[0]))) {
   say sprintf("%04X", $ucp_num);
}

Demo:

$ ./a2.pl aé€♠𠀀
0061
00E9
20AC
2660
20000

---

To get the characters of a string as strings:

• unpack('(a)*', $_)
• split(//, $_)

To get the characters of a string as numbers:

• unpack('W*', $_)
• map { ord($_) } split(//, $_))

To convert a string of bytes (characters in range 0x00..0xFF) into hex:

• unpack('H*', $_)
• join "", map { sprintf('%02X', $_) } split(//, $_))

Easy way to see the characters of a string as hex for debugging:

• sprintf("%vX", $_)