How can I replace non-printable Unicode characters in Java?

Question

The following will replace ASCII control characters (shorthand for [\x00-\x1F\x7F]):

my_string.replaceAll("\\p{Cntrl}", "?");

The following will replace all ASCII non-printable characters (shorthand for [\p{Graph}\x20]), including accented characters:

my_string.replaceAll("[^\\p{Print}]", "?");

However, neither works for Unicode strings. Does anyone has a good way to remove non-printable characters from a unicode string?

Answer 1

my_string.replaceAll("\\p{C}", "?");

See more about Unicode regex. java.util.regexPattern/String.replaceAll supports them.

Answer 2

Op De Cirkel is mostly right. His suggestion will work in most cases:

myString.replaceAll("\\p{C}", "?");

But if myString might contain non-BMP codepoints then it's more complicated. \p{C} contains the surrogate codepoints of \p{Cs}. The replacement method above will corrupt non-BMP codepoints by sometimes replacing only half of the surrogate pair. It's possible this is a Java bug rather than intended behavior.

Using the other constituent categories is an option:

myString.replaceAll("[\\p{Cc}\\p{Cf}\\p{Co}\\p{Cn}]", "?");

However, solitary surrogate characters not part of a pair (each surrogate character has an assigned codepoint) will not be removed. A non-regex approach is the only way I know to properly handle \p{C}:

StringBuilder newString = new StringBuilder(myString.length());
for (int offset = 0; offset < myString.length();)
{
    int codePoint = myString.codePointAt(offset);
    offset += Character.charCount(codePoint);

    // Replace invisible control characters and unused code points
    switch (Character.getType(codePoint))
    {
        case Character.CONTROL:     // \p{Cc}
        case Character.FORMAT:      // \p{Cf}
        case Character.PRIVATE_USE: // \p{Co}
        case Character.SURROGATE:   // \p{Cs}
        case Character.UNASSIGNED:  // \p{Cn}
            newString.append('?');
            break;
        default:
            newString.append(Character.toChars(codePoint));
            break;
    }
}

Answer 3

methods below for your goal

public static String removeNonAscii(String str)
{
    return str.replaceAll("[^\\x00-\\x7F]", "");
}

public static String removeNonPrintable(String str) // All Control Char
{
    return str.replaceAll("[\\p{C}]", "");
}

public static String removeSomeControlChar(String str) // Some Control Char
{
    return str.replaceAll("[\\p{Cntrl}\\p{Cc}\\p{Cf}\\p{Co}\\p{Cn}]", "");
}

public static String removeFullControlChar(String str)
{
    return removeNonPrintable(str).replaceAll("[\\r\\n\\t]", "");
}