Java support for non-BMP Unicode characters (i.e. codepoints > 0xFFFF) in their Regular Expression Library?

Question

I'm currently using Java 6 (I don't have the option of moving to Java 7) and I'm trying to use the java.util.regex package to do pattern matching of strings that contain Unicode characters.

I know that java.lang.String supports supplemental characters (i.e. characters with codepoints > 0xFFFF) (since Java 5), but I don't see a simple way to do do pattern matching with these characters. java.util.regex.Pattern still only allows hexadecimals to be represented using 4 digits (e.g. \uFFFF)

Does anyone know if I'm missing an API here?

Answer 1

The easiest solution is to use a UTF-8 encoding for your source code. Then just put the characters in directly. You should never ever ever have to specify separate code units in any program.

There is still an issue with character classes, however, because Java’s lamely exposed UTF-16 internal encoding messes them up. You can’t use full Unicode until JDK7, where even then you will have to specify logical code points using an indirect \x{HHHHH} notation. You still won’t be able to have any literal code point in a charclass, but you can dodge it with \x{H..H}.

Imperfect, but it’s a lot better than it was. UTF-16 is always a compromise. Systems that use UTF-8 or UTF-32 internally don’t have these restrictions. They also never make you specify code units that aren’t identical to code points.

Answer 2

I've never done pattern matching with supplemental characters, but I think it's as simple as encoding them (in patterns and strings) as two 16 bits numbers (a UTF-16 surrogate pair) \unnnn\ummmm . java.util.regex should be is clever enough to interpret those two numbers (Java chars) as a single character in patterns and strings (though Java will still see them as two chars, as elements of the string).

Two links:

Java Unicode encoding

http://java.sun.com/developer/technicalArticles/Intl/Supplementary/

From the last link (refering to Java 5) :

The java.util.regex package has been updated so that both pattern strings and target strings can contain supplementary characters, which will be handled as complete units.

Note also that, if you are using UTF8 as your encoding (for your source files), you can also write them directly (see section "Representing Supplementary Characters in Source Files" in the last link).

For example:

    String pat1 = ".*\uD840\uDC00{2}.*";
    String s1  = "HI \uD840\uDC00\uD840\uDC00 BYE";
    System.out.println(s1.matches(pat1) + " len=" + s1.length());

    String pat2 = ".*\u0040\u0041{2}.*";
    String s2 = "HI \u0040\u0041\u0040\u0041 BYE";
    System.out.println(s2.matches(pat2) + " len=" + s2.length());

This, compiled with Java 6, prints

true len=11
false len=11

which agrees with the above. In the first case, we have a single code point, represented as a pair of surrogate java chars (two 16 bits chars, one suplemental Unicode character), and the {2} quantifier applies to the pair(=codepoint). In the second, we have two distinct BMP characters, the quantifier applies to the last one - hence, no match.

Notice, however, that the string length is the same (because Java measures the string length counting Java characters, not Unicode code points).