Why does 行ける parse into a single token, but 見られる parses into 2(見+られる)?

Question

Both represent the same form of different types of verbs - shouldn't they both parse into a single token? Even if 2 tokens makes more sense, they should be consistent and both parse into 2 I would think.

Edit: it was pointed out in comments that 見られる could also be passive - another example would be 食べれる, which also tokenizes into 2 tokens (食べ+れる), but is certainly potential.

Answer 1

Short answer: because 行ける/見える are in the dictionary and 見られる isn't. (note: this is the case for both ipadic and unidic.)

In the case of 行ける and 見られる the distinction is pretty simple - 行く is a 五段/five-step verb and 見る is not. In Unidic and IPAdic stems of five-step verbs are registered due to the way verb endings are treated. Verb endings are basically all 助動詞 (recognizable units like られる that can't stand alone) or 補助動詞 (things like しまう that can stand alone), but dictionary-form endings like る or う aren't considered as either of those so they don't get their own part of speech tag and form one token with the verb root.

With 見える the situation is a little trickier - 見える is being treated as a root verb, and not just as the potential form of 見る. If you look at lex.csv in Unidic for example you'll see a bunch of conjugations of 見える where 見える is given as the base form. Looking at dictionaries it seems common for 見える to have its own entry, partly for historical reasons (check まみえる).

For a longer explanation of how and why verbs are broken into multiple tokens, look up the details of Short Unit Words, Medium Unit Words, Long Unit Words, and Bunsetsu. Documentation from NINJAL covers the concepts but with little detail for verbs; Comainu is a system that can detect all of these classes; and this lengthy article provides a good overview of the history in English.

Hope that helps!