Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

removing newlines from messy strings in pandas dataframe cells?

I've used multiple ways of splitting and stripping the strings in my pandas dataframe to remove all the '\n'characters, but for some reason it simply doesn't want to delete the characters that are attached to other words, even though I split them. I have a pandas dataframe with a column that captures text from web pages using Beautifulsoup. The text has been cleaned a bit already by beautifulsoup, but it failed in removing the newlines attached to other characters. My strings look a bit like this:

"hands-on\ndevelopment of games. We will study a variety of software technologies\nrelevant to games including programming languages, scripting\nlanguages, operating systems, file systems, networks, simulation\nengines, and multi-media design systems. We will also study some of\nthe underlying scientific concepts from computer science and related\nfields including"

Is there an easy python way to remove these "\n" characters?

Thanks in advance!

like image 215
Calvin Avatar asked May 28 '17 13:05

Calvin


2 Answers

EDIT: the correct answer to this is:

df = df.replace(r'\n',' ', regex=True)  

I think you need replace:

df = df.replace('\n','', regex=True) 

Or:

df = df.replace('\n',' ', regex=True) 

Or:

df = df.replace(r'\\n',' ', regex=True) 

Sample:

text = '''hands-on\ndev nologies\nrelevant scripting\nlang ''' df = pd.DataFrame({'A':[text]}) print (df)                                                    A 0  hands-on\ndev nologies\nrelevant scripting\nla...  df = df.replace('\n',' ', regex=True) print (df)                                                 A 0  hands-on dev nologies relevant scripting lang  
like image 198
jezrael Avatar answered Sep 24 '22 12:09

jezrael


df.replace(to_replace=[r"\\t|\\n|\\r", "\t|\n|\r"], value=["",""], regex=True, inplace=True) 

worked for me.

Source:

https://gist.github.com/smram/d6ded3c9028272360eb65bcab564a18a

like image 23
LinuxUser Avatar answered Sep 25 '22 12:09

LinuxUser