Faster Lemmatization techniques in Python

Question

I am trying to find out a faster way to lemmatize words in a list (named text) using the NLTK Word Net Lemmatizer. Apparently this is the most time consuming step in my whole program(used cProfiler to find the same).

Following is the piece of code that I am trying to optimize for speed -

def lemmed(text):
    l = len(text)
    i = 0
    wnl = WordNetLemmatizer()
    while (i<l):
        text[i] = wnl.lemmatize(text[i])
        i = i + 1
    return text

Using the lemmatizer decreases my performance by 20x. Any help would be appreciated.

Answer 1

If you have a few cores to spare, try using the multiprocessing library:

from nltk import WordNetLemmatizer
from multiprocessing import Pool

def lemmed(text, cores=6): # tweak cores as needed
    with Pool(processes=cores) as pool:
        wnl = WordNetLemmatizer()
        result = pool.map(wnl.lemmatize, text)
    return result


sample_text = ['tests', 'friends', 'hello'] * (10 ** 6)

lemmed_text = lemmed(sample_text)

assert len(sample_text) == len(lemmed_text) == (10 ** 6) * 3

print(lemmed_text[:3])
# => ['test', 'friend', 'hello']