Should I perform both lemmatization and stemming?

Question

I'm writing a text classification system in Python. This is what I'm doing to canonicalize each token:

lem, stem = WordNetLemmatizer(), PorterStemmer()
for doc in corpus:
    for word in doc:
        lemma = stem.stem(lem.lemmatize(word))

The reason I don't want to just lemmatize is because I noticed that WordNetLemmatizer wasn't handling some common inflections. In the case of adverbs, for example, lem.lemmatize('walking') returns walking.

Is it wise to perform both stemming and lemmatization? Or is it redundant? Do researchers typically do one or the other, and not both?

Answer 1

From my point of view, doing both stemming and lemmatization or only one will result in really SLIGHT differences, but I recommend for use just stemming because lemmatization sometimes need 'pos' to perform more presicsely.

For example, if you want to lemmatize "better", you should explicitly indicate pos: print(lemmatizer.lemmatize("better", pos="a"))

If not supplied, the default is "noun"

Answer 2

The lemmatization of walking is ambiguous. Walking, when used as an adjective, is its own baseform (rather than walk).

Correction: Research has shown that generally stemming outperforms lemmatization in IR tasks. A qualitative comparison between the two and an explanation can be found here.