How to detect language of text?

Question

I have a form which lets users input text snippets. So how can figure out the language of the entered text?

Specifically these languages for now:

Arabic: هذه هي بعض النصوص العربية

Chinese: 这是一些阿拉伯文字

Japanese: これは、いくつかのアラビア語のテキストです

[Edit] The detection has work on text which is retrieved via an API too (no browser involved)

Answer 1

You can figure out whether the characters are from the Arabic, Chinese, or Japanese sections of the Unicode map.

If you look at the list on Wikipedia, you'll see that each of those languages has many sections of the map. But you're not doing translation, so you don't need to worry about every last glyph.

For example, your Chinese text begins (in hex) 0x8FD9 0x662F 0x4E00 - and those are all in the "CJK Unified Ideographs" section, which is Chinese. Here are a few ranges to get you started:

Arabic (0600–06FF)

Japanese

• Hiragana (3040–309F)
• Katakana (30A0–30FF)
• Kanbun (3190–319F)

Chinese

• CJK Unified Ideographs (4E00–9FFF)

(I got the hex for your Chinese by using a Chinese to Unicode Converter.)