Convert optically equivalent unicode strings to ASCII in Java?

Question

I run a social network that requires unicode usernames to be unique (as expected).

Some creative users have started using Cyrillic (and other) unicode characters to create optically equivalent (but unicode distinct) usernames.

For example, they'll use the Cyrillic small letter a 'а', which looks identical to the roman one.

Does anyone know of a way to convert these optically equivalent characters automatically in Java? I'd rather not have to create a conversion table by hand if a mechanism already exists.

Answer 1

You can try Unicode normalization - basically, indistinguishable code points have a 'canonical' code point designated, and normalization is the process of replacing each character with its canonical form.

Java seems to support Unicode normalization via java.text.Normalizer - more info here.

However, I'm not sure that latin A and cyrillic A are marked as equivalent in Unicode - you'd have to try.

This will also not help you when your users start using very similar instead of identical characters - humans are very inventive and a technical solution might not work 100% here, so you will probably have to resort to human moderation anyway.

There are also some other solutions - limiting the usernames to latin alphanumerics, for example.

Answer 2

Why don't you try to apply an OCR library.