Java regex working differently on Android than in Java

Question

I'm using Java regex on Android and I'm seeing strange differences, as the following

Java: "COSÌ".replaceAll( "\\W", "" ) ----> "COS"

Android: "COSÌ".replaceAll( "\\W", "" ) ----> "COSÌ"

Anyone noticed similar differences between Java and Android String class?

Answer 1

Android

Straight from the Android documentation, right after the list of short-hand character classes (\d, \w, \s, etc.):

Note that these built-in classes don't just cover the traditional ASCII range. For example, \w is equivalent to the character class [\p{Ll}\p{Lu}\p{Lt}\p{Lo}\p{Nd}].

This would also explain why Ì is not replaced for the same code running on Android version.

While it is correct that the short-hand character classes also match Unicode character, the sample definition of \w Android documentation is way outdated. See Appendix for more details.

Java SE

In contrast, in Java SE, by default, \w is equivalent to [a-zA-Z_0-9].

\w only matches Unicode word character when Pattern.UNICODE_CHARACTER_CLASS flag is specified. When the flag is specified:

• In Java 7, \w has the same definition as [\p{IsAlphabetic}\p{M}\p{Nd}\p{Pc}]
• In Java 8, \w is updated to [\p{IsAlphabetic}\p{M}\p{Nd}\p{Pc}\u200c\u200d]

Workaround

Specify the character class directly. ICU regex doesn't support ASCII character class:

[^a-zA-Z0-9_]

---

Appendix

Definition of \w in ICU

Here is the how the \w has evolved over time:

• The short-hand character class \w was defined as [\p{Ll}\p{Lu}\p{Lt}\p{Lo}\p{Nd}] (as shown in the documentation) up to ICU 3.0.

• From ICU 3.2 (released on 2006/02/24) and up to and including ICU 4.8.1.1, [\p{Alphabetic}\p{Mark}\p{Decimal_Number}\p{Connector_Punctuation}] (equivalent to [\p{Alphabetic}\p{M}\p{Nd}\p{Pc}] in the source code) is used instead. Changed at revision 16634

• From ICU 49 (released on 2012/06/06), the current definition in the documentation is used [\p{Alphabetic}\p{Mark}\p{Decimal_Number}\p{Connector_Punctuation}\u200c\u200d] (equivalent to [\p{Alphabetic}\p{M}\p{Nd}\p{Pc}\u200c\u200d] in the source code). Changed at revision 31278.

The string above is used to construct URX_ISWORD_SET, which is used in regcmp.cpp in doBackslashW to compile the regex.

ICU version used by Android

Even at android-1.6_r1 (Donut), when Pattern class documentation is barren, it is already using ICU 3.8. The source code shows that it is using the definition from the second bullet point.

The documentation probably falls back to describe the behavior of the oldest version of Android.

Reference

If you want to navigate around the source code of Android yourself:

• libcore (Java Class Library)

  • From android-1.6_r1 up to android-2.2.3_r2.1, platform/dalvik repository. Pattern class can be located at libcore/regex/src/main/java/java/util/regex/Pattern.java
  • From android-2.3_r1 to now, platform/libcore repository. Pattern class can be located at /luni/src/main/java/java/util/regex/Pattern.java

• icu4c (ICU library for C)

  • From android-1.6_r1 up to android-4.4.4_r2.0.1, platform/external/icu4c repository. Regex related stuffs can be found in i18n, Unicode related stuffs can be found in common.
  • From android-5.0.0_r1 to now, platform/external/icu. Enter icu4c/source, then similar path as above.