How to apply Polyglot Detector function to dataframe

Question

Assuming I have a column called df.Text which contains text (more that 1 sentence) and I want to use polyglot Detector to detect the language and store the value in a new column df['Text-Lang'] how do I ensure I also capture the other details like code and confidence

testEng ="This is English"
lang = Detector(testEng)
print(lang.language)

returns

name: English code: en confidence: 94.0 read bytes: 1920

but

df['Text-Lang','Text-LangConfidence']= df.Text.apply(Detector)

ends with

AttributeError: 'float' object has no attribute 'encode' and Detector is not able to detect the language reliably.

Am I applying the Detector function incorrectly or storing the output incorrectly or something else?

Answer 1

First, if you only need polyglot for language detection, you'd better use pycld2 directly, that is what used behind the scenes. It has much cleaner API.

Saying that, the error you state comes from one of the values in your Text column, which is a real number. So you will have to convert values like that into strings.

The next problem you will stumble upon is minimal text length. polyglot will throw exception if the text is too short. You have to silence the exception by passing quiet=True.

Now, applying Detector will return an object. So you will have to parse it to extract the information you want. To extract language names, you will have to import icu module (it is a dependency of polyglot, so you have it installed already):

import icu
df.Text = df.Text.astype(str)
df['poly_obj'] = df.Text.apply(lambda x: Detector(x, quiet=True))
df['Text-lang'] = df['poly_obj'].apply(lambda x: icu.Locale.getDisplayName(x.language.locale))
df['Text-LangConfidence'] = df['poly_obj'].apply( lambda x: x.language.confidence)

After that you can drop the poly_obj column.