Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Using awk to detect UTF-8 multibyte character

Tags:

awk

I am using awk (symlinked to gawk on my machine) to read through a file and get a character count per line to test if a file is fixed width. I can then re-use the following script with the -b --characters-as-bytes option to see if the file is fixed width by byte.

#!/usr/bin/awk -f

BEGIN {
    width = -1;
}

{
    len = length($0);

    if (width == -1) {
        width = len;
    } else if (len != 0 && len != width) {
        exit 1;
    }
}

I want to do something similar to test whether each line in a file has the same amount of bytes and characters to assume all characters are a single byte (I do realize this is subject false negatives). The challenge is that I would like to run through the file one time and break out at first mismatch. Is there a way to set the -b option from within an awk script similar to how you can adjust FS. If this isn't possible, I'm open to options outside of awk. I can always just write this in C if I have to, but I wanted to make sure there isn't something already available.

Efficiency is what I am aiming for here. Having this information will help me skip a costly process, so I don't this in itself to be costly. I'm dealing with files that can be over 100 million lines long.

Clarification

I want something like the above. Something like this

#!/usr/bin/awk -f
{
    if (length($0) != bytelength($0))
        exit 1;
}

I don't need any output. I will just trigger off the return code ($? in bash). So exit 1 if this fails. Obviously bytelength is not a function. I'm just looking for a way to achieve this without running awk twice.

UPDATE

sundeep's solution works for what I have described above:

awk -F '' -l ordchr '{for(i=1;i<=NF;i++) if(ord($i)<0) {exit 1;}}'

I was operating under the assumption that awk would count a higher-end character with a Windows single-byte encoding above 0x7F as a single character, but it actually doesn't count it at all. So byte length would still not be the same as length. I guess I will need to write this in C for something that specific.

Conclusion

So I think I did a poor job of explaining my problem. I receive data that is encoded in either UTF-8 or Windows' style single-byte encoding like CP1252. I wanted to check if there are any multibyte characters in the file and exit if found. I originally wanted to do this in awk, but I playing with files that may have a different encoding has proven difficult.

So in a nutshell if we assume a file with a single character in it:

CHARACTER  FILE_ENCODING     ALL_SINGLE_BYTE   IN_HEX
á          UTF-8             false             0xC3 0xA1
á          CP1252            true              0xE1
a          ANY               true              0x61
like image 267
Jason Avatar asked Jul 30 '26 23:07

Jason


1 Answers

You seem to be targeting UTF-8 specifically. Indeed first multibyte character in UTF-8 encoding starts 0b11xxxxxx and the next byte needs to be 0b10xxxxxx where x represents any value (from wikipedia).

So you can detect such sequence with sed by matching the hex ranges and exit with nonzero exit status if found:

LC_ALL=C sed -n '/[\xC0-\xFF][\x80-\xBF]/q1'

Ie. match bytes in ranges [0b11000000-0b11111111][0b10000000-0b10111111].

I think \x?? and q are both GNU extensions to sed.

like image 93
KamilCuk Avatar answered Aug 02 '26 20:08

KamilCuk