Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Detect if character is simplified or traditional Chinese character

Tags:

unicode

cjk

I found this question which gives me the ability to check if a string contains a Chinese character. I'm not sure if the unicode ranges are correct but they seem to return false for Japanese and Korean and true for Chinese.

What it doesn't do is tell if the character is traditional or simplified Chinese. How would you go about finding this out?


update

Q: How can I recognize from the 32 bit value of a Unicode character if this is a Chinese, Korean or Japanese character?

http://unicode.org/faq/han_cjk.html

Their argument that the characters regardless of their shape have the same meaning and therefore should be represented by the same code. Well, it's not meaningless to me because I am analyzing individual characters which doesn't work with their solution:

A better solution is to look at the text as a whole: if there's a fair amount of kana, it's probably Japanese, and if there's a fair amount of hangul, it's probably Korean.

like image 761
thenengah Avatar asked Jan 06 '11 20:01

thenengah


People also ask

How do you know if a text is simplified or Traditional Chinese?

The most obvious difference between traditional Chinese and simplified Chinese is the way that the characters look. Traditional characters are typically more complicated and have more strokes, while simplified characters are, as the name suggests, simpler and have fewer strokes.

How do I check my Chinese characters?

Optical character recognition (OCR) – Many apps and websites provide OCR features where you can scan or take pictures of the character(s) you want to look up. Google Docs has such a feature and there are others online you can easily find by searching for “Chinese” and “OCR”.

Should I use traditional or simplified Chinese?

Geography will help us here: Simplified Chinese is used in mainland China (PRC) and in Singapore. If you are targeting Hong Kong, Taiwan, or Malaysia your choice should be to use the Traditional Chinese.


2 Answers

As already stated, you can't reliably detect the script style from a single character, but it is possible for a sufficiently long sample of text. See https://github.com/jpatokal/script_detector for a Ruby gem that does the job, and Simplified Chinese Unicode table for a general discussion.

like image 92
lambshaanxy Avatar answered Oct 15 '22 11:10

lambshaanxy


It is possible for some characters. The Traditional and Simplified character sets overlap, so you have basically three sets of characters:

  1. Characters that are traditional only.
  2. Characters that are simplified only.
  3. Characters that have been left untouched, and are available in both.

Take the character 面 for instance. It belongs both to #2 and #3... As a simplified character, it stands for 面 and 麵, face and noodles. Whereas 麵 is a traditional character only. So in the Unihan database, 麵 has a kSimplifiedVariant, which points to 面. So you can deduct that it is a traditional character only.

But 面 also has a kTraditionalVariant, which points to 麵. This is where the system breaks: if you use this data to deduct that 面 is a simplified character only, you'd be wrong...

On the other hand, 韩 has a kTraditionalVariant, pointing to 韓, and these two are a "real" Simplified/Traditional pair. But nothing in the Unihan database differentiates cases like 韓/韩 from cases like 麵/面.

like image 20
dda Avatar answered Oct 15 '22 10:10

dda