ICU: Transliterate and then remove all non-alphanumeric characters

Question

Can it be done with ICU without falling back to regex?

Currently I normalize filenames like this:

protected function normalizeFilename($filename)
{
    $transliterator = Transliterator::createFromRules(
        'Any-Latin; Latin-ASCII; [:Punctuation:] Remove;'
    );
    $filename = $transliterator->transliterate($filename);
    $filename = preg_replace('/[^A-Za-z0-9_]/', '', $filename);
    return $filename;

}

Can I get rid of regular expression here and do everything with ICU calls?

Answer 1

Use the correct tool for the job

I don't see anything wrong with what you're doing now.

ICU transliteration is first and foremost language oriented. It tries to preserve meaning.

Regular expressions, on the other hand, can manipulate characters in detail, giving you the assurance that the file name is restricted to the selected characters.

The combination is perfect, in this case.

I have, of course, looked for a solution to your question. But to be honest, I couldn't find something that would work on all possible inputs.

For instance, not all characters, we would consider punctuation marks, are removed by [:Punctuation:] Remove;. Try the Russian name: Корнильев, Кирилл. After applying your id it becomes: Kornilʹev Kirill. Clearly that's not a punctuation mark, but you don't want it in your file name.

So I would advice to use the correct tool for the job:

1. Use ICU to get the best ASCII enquivalent. Only using Latin-ASCII; as the id will do. Nice and simple.
2. Then use a regular expression, just like you did, to make sure you're left with only the characters you need.

There is really nothing wrong with this.

PS: Personally I think the person, or persons, who wrote the ICU user guide should not be complimented on a job well done. What a mess.