How to split strings inside a list by whitespace characters

Question

So stdin returns a string of text into a list, and multiple lines of text are all list elements. How do you split them all into single words?

mylist = ['this is a string of text \n', 'this is a different string of text \n', 'and for good measure here is another one \n']

wanted output:

newlist = ['this', 'is', 'a', 'string', 'of', 'text', 'this', 'is', 'a', 'different', 'string', 'of', 'text', 'and', 'for', 'good', 'measure', 'here', 'is', 'another', 'one']

Answer 1

You can use simple list comprehension, like:

newlist = [word for line in mylist for word in line.split()]

This generates:

>>> [word for line in mylist for word in line.split()]
['this', 'is', 'a', 'string', 'of', 'text', 'this', 'is', 'a', 'different', 'string', 'of', 'text', 'and', 'for', 'good', 'measure', 'here', 'is', 'another', 'one']

Answer 2

You could just do:

words = str(list).split()

So you turn the list into a string then split it by a space bar. Then you can remove the /n's by doing:

words.replace("/n", "")

Or if you want to do it in one line:

words = str(str(str(list).split()).replace("/n", "")).split()

Just saying this may not work in python 2

Answer 3

Alternatively, you can map str.split method to every string inside the list and then chain the elements from the resulting lists together by itertools.chain.from_iterable:

from itertools import chain

mylist = ['this is a string of text \n', 'this is a different string of text \n', 'and for good measure here is another one \n']
result = list(chain.from_iterable(map(str.split, mylist)))
print(result)
# ['this', 'is', 'a', 'string', 'of', 'text', 'this', 'is', 'a', 'different', 'string', 'of', 'text', 'and', 'for', 'good', 'measure', 'here', 'is', 'another', 'one']

Answer 4

Besides the list comprehension answer above that i vouch for, you could also do it in a for loop:

#Define the newlist as an empty list
newlist = list()
#Iterate over mylist items
for item in mylist:
 #split the element string into a list of words
 itemWords = item.split()
 #extend newlist to include all itemWords
 newlist.extend(itemWords)
print(newlist)

eventually your newlist will contain all split words that were in all elements in mylist

But the python list comprehension looks much nicer and you can do awesome things with it. Check here for more:

https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions