Text Language detection in python

Question

I am trying to detect the language of the text that may consist of an unknown number of languages. The following code gives me different languages as answer NOTE: I reduced the review becuase it was giving the error during post "" are not allowed

print(detect(كانت جميله وممتعة للأطفال اولا حيث اماكن اللعبر))
print(detect(的马来西亚))
print(detect(Vi havde 2 perfekte dage i Legoland Malaysia))
print(detect(Wij hebben alleen gekozen voor het waterpark maar daar ben je vrijs snel doorheen. Super leuke glijbanen en overal ruimte om te zitten en te liggen. Misschien volgende keer een gecombineerd ticket kopen met ook toegang tot waterpark))
print(detect(This is a park thats just ok, nothing great to write home about.  There is barely any shade, the weather is always really hot so they need to take this into consideration. The atractions are just meh. I would only go if you are a fan of lego, for the sculptures are nice.))

Here is the output

ar
zh-cn
da
nl
en

But using the following loop, all reviews give me 'en' as result

from langdetect import detect
import pandas as pd
df = pd.read_excel('data.xls') #
lang = []    
for r in df.Review:
    lang = detect(r)
    df['Languagereveiw'] = lang

the output is 'en' for all five rows.

Need guidance that where is the missing chain?

Here is the sample data

Secondly, How can I get the complete name of languages i.e. English for 'en'

Answer 1

In your loop you're overwriting the entire column by doing this:

df['Languagereveiw'] = lang

If you want to do this in a for loop use iteritems:

for index, row in df['Review'].iteritems():
    lang = detect(row) #detecting each row
    df.loc[index, 'Languagereveiw'] = lang

however, you can just ditch the loop and just do

df['Languagereveiw'] = df['Review'].apply(detect)

Which is syntactic sugar to execute your func on the entire column

Regarding your latter question about converting from language code to full description:

'en' to 'english',

look at polyglot

this provides the facility to detect language, get the language code, and the full description

Answer 2

You can try this: First, let us create a data frame that contains text written in different languages:

import pandas as pd
from langdetect import detect

df = pd.DataFrame({'text': ['This is written in English', 'هذا مكتوب باللغة العربية', 'English is easy to learn']})
df.head()

    text
0   This is written in English
1   هذا مكتوب باللغة العربية
2   English is easy to learn

Then, the following function will be used to get the language of each text:

def detect_lang(text):
    return detect(text)

df['text'].apply(detect_lang)

output:

0    en
1    ar
2    en

If you would like to filter for the English language only, you could use this function:

def detect_en(text):
    try:
        return detect(text) == 'en'
    except:
        return False

df = df[df['text'].apply(detect_en)]
df

output:

    text
0   This is written in English
2   English is easy to learn

Enjoy!