How does tokenization and pattern matching work in Chinese.?

Question

This question involves computing as well as knowledge of Chinese. I have chinese queries and I have a separate list of phrases in Chinese I need to be able to find which of these queries have any of these phrases.

In english, it is a very simple task. I don't understand Chinese at all, its semantics, grammar rules etc. and if somebody in this forum who also understands Chinese can help me with some basic understanding and how pattern matching is done for Chinese.

I have a basic perception that in Chinese one unit (without any space in between) can actually mean more than one word(Is this correct?). So are there any rules on how more than one word combine among themselves to stand out as a unit. It is confusing because there are spaces in Chinese writing yet even a unit without space has more than one word in it.

Any links which explain Chinese from computational point of view, pattern matching etc would be very useful..

Answer 1

I have a basic perception that in Chinese one unit (without any space in between) can actually mean more than one word(Is this correct?).

In Chinese spaces are rarely used, eg:

递归（英语：Recursion），又譯為遞迴，在数学与计算机科学中，是指在函数的定义中使用函数自身的方法。递归一词还较常用于描述以自相似方法重复事物的过程。例如，当两面镜子相互之间近似平行时，镜中嵌套的图像是以无限递归的形式出现的。

You'll notice what appear to be spaces actually are just Chinese punctuation characters, which just have more padding than usual.

So are there any rules on how more than one word combine among themselves to stand out as a unit. It is confusing because there are spaces in Chinese writing yet even a unit without space has more than one word in it.

Think of it this way: one Chinese character is very, very roughly similar to one English word. Often times two or more characters need to be combined to form one word, and each separate character may mean something completely different depending on context.

To meaningfully tokenize Chinese text you'd have to segment words taking that in consideration.

See Chinese Natural Language Processing and Speech Processing, from the Stanford NLP group.