Regular expression for zero-width Unicode characters

Question

I’m converting a table to a human-readable form, and I need to treat the content with zero-length Unicode characters “in a distinctive way”.

The only way to recognize them seems to be (even with Perl’s extensions!) to list all the possibilities “which get in the way” (what I found in my tables are CGJ, ZW [non]joiners, and r2l/l2r marks). I tried to use Perl’s `\p{XPosixPrint}, but it fails:

perl -we "print(qq(\n)), print, print qq(\t), (chr hex) =~ /\p{XPosixPrint}/ for @ARGV" 034f 200c 200d 200e 200f

034f    1
200c    1
200d    1
200e    1
200f    1

Did I miss anything: is there a (combination of) \p{…} which would match zero-width Unicode characters (and only them)?

Answer 1

No, there is no character property in the Unicode standard specifically for that purpose, but a good approximation of what you’re after may be Default_Ignorable_Code_Point, as defined in the DerivedCoreProperties.txt data file. This property set includes:

• Most format control characters that regularly do not have a glyphic appearance of their own (such as Zero Width Space, the join controls, and the bidirectional controls)
• Most invisible combining marks (such as CGJ and variation selectors)
• Miscellaneous characters like the Hangul jamo fillers, which typically have neither glyph nor advance width
• Ranges of currently unassigned code points that are informally reserved for future additions of similar nature

Being zero-width is not a defined concept in the Unicode standard, so you will likely still need to make some adjustments to the set of characters depending on your exact needs. For example, the “interlinear annotation” format controls (U+FFF9..U+FFFB) are deliberately excluded from the Default_Ignorable_Code_Point property despite fitting your criteria for zero-width characters quite well.

The purpose of Default_Ignorable_Code_Point is to mark characters that, if not supported, should be discarded entirely when rendering text instead of being substituted with a placeholder glyph (such as a box or a question mark) as would be common practice for most other characters. Naturally this includes a lot of characters that are meant to be invisible in the first place.

However, some of these invisible characters are really important for parsing the structure of the text they appear in. If they’re not properly supported it is actually best practice to represent them with a visible placeholder so that the reader at least knows that something went wrong on their end. This is why characters like the aforementioned interlinear annotation controls are explicitly not default-ignorable. The C0 and C1 controls (gc=Cc, i.e. U+0000..U+001F and U+007F..U+009F) are not default-ignorable for the same reason.

There are other edge cases that cannot be resolved easily. For example, whether or not U+1D159 MUSICAL SYMBOL NULL NOTEHEAD is invisible and zero-width depends entirely on the whims of the font currently being used to display it. There also exist so-called “invisible stackers” (Indic_Syllabic_Category=Invisible_Stacker) in many writing systems which are zero-width combining marks that modify the appearance of adjacent characters without being visible themselves except when they’re being used in an invalid or incomplete sequence, in which case they may or may not show up depending on the font and the text renderer. None of these characters are default-ignorable.

If your goal is to catch 100% of characters that “get in the way” according to your personal criteria, you will have to keep a custom list of characters and be ready to update it with every new Unicode release. If 99% is good enough, I’d recommend simply relying on the Default_Ignorable_Code_Point property with a few hard-coded additions like the interlinear annotation stuff and the C0 and C1 sets. Most of these characters discussed here are incredibly obscure and unlikely to ever show up in normal text data.

Answer 2

The earlier answer focuses on the choice of characters to include, so I'm going to focus on how to match those characters if you choose to go with a custom list.

Using [\x{200B}-\x{200D}\x{FEFF}] as an example,

• You could define your own class.

  sub IsZW {
     state $IsZW =
        join "\n", map s/-/ /r, split /,/,
           "200B-200D,FEFF";
  
     $IsZW
  }
  
  /...\p{ IsZW }.../
  

• You could interpolate a subpattern.

  my $zw_char = qr/[\x{200B}-\x{200D}\x{FEFF}]/;
  
  /...$zw_char.../
  

• You could define a subpattern.

  /
     ...(?&zw_char)...
  
     (?(DEFINE)
        (?<zw_char> [\x{200B}-\x{200D}\x{FEFF}] )
     )
  /x
  

So, once you decide on the list of characters in which you are interested, there are plenty of options to make it easy to use.