removing unicode from text in pandas

Question

for one string, the code below removes unicode characters & new lines/carriage returns:

t = "We've\xe5\xcabeen invited to attend TEDxTeen, an independently organized TED event focused on encouraging youth to find \x89\xdb\xcfsimply irresistible\x89\xdb\x9d solutions to the complex issues we face every day.,"

t2 = t.decode('unicode_escape').encode('ascii', 'ignore').strip()
import sys
sys.stdout.write(t2.strip('\n\r'))

but when I try to write a function in pandas to apply this to every cell of a column, it either fails because of an attribute error or I get a warning that a value is trying to be set on a copy of a slice from a DataFrame

def clean_text(row):
    row= row["text"].decode('unicode_escape').encode('ascii', 'ignore')#.strip()
    import sys
    sys.stdout.write(row.strip('\n\r'))
    return row

applied to my dataframe:

df["text"] = df.apply(clean_text, axis=1)

how can I apply this code to each element of a Series?

Answer 1

The problem seems like you are trying to access and alter row['text'] and return the row itself when doing the apply function, when you do apply on a DataFrame, it's applying to each Series, so if changed to this should help:

import pandas as pd

df = pd.DataFrame([t for _ in range(5)], columns=['text'])

df 
                                                text
0  We've������been invited to attend TEDxTeen, an ind...
1  We've������been invited to attend TEDxTeen, an ind...
2  We've������been invited to attend TEDxTeen, an ind...
3  We've������been invited to attend TEDxTeen, an ind...
4  We've������been invited to attend TEDxTeen, an ind...

---

def clean_text(row):
    # return the list of decoded cell in the Series instead 
    return [r.decode('unicode_escape').encode('ascii', 'ignore') for r in row]

df['text'] = df.apply(clean_text)

df
                                                text
0  We'vebeen invited to attend TEDxTeen, an indep...
1  We'vebeen invited to attend TEDxTeen, an indep...
2  We'vebeen invited to attend TEDxTeen, an indep...
3  We'vebeen invited to attend TEDxTeen, an indep...
4  We'vebeen invited to attend TEDxTeen, an indep...

Alternatively you might use lambda as below, and directly apply to only text column:

df['text'] = df['text'].apply(lambda x: x.decode('unicode_escape').\
                                          encode('ascii', 'ignore').\
                                          strip())

Answer 2

I actually can't reproduce your error: the following code runs for me without an error or warning.

df = pd.DataFrame([t,t,t],columns = ['text'])
df["text"] = df.apply(clean_text, axis=1)

If it helps, I think a more "pandas" way to approach this type of problem might be to use a regex with one of the DataFrame.str methods for example:

df["text"] =  df.text.str.replace('[^\x00-\x7F]','')