Partitioning a string in Python by a regular expression

Question

I need to split a string into an array on word boundaries (whitespace) while maintaining the whitespace.

For example:

'this is  a\nsentence'

Would become

['this', ' ', 'is', '  ', 'a' '\n', 'sentence']

I know about str.partition and re.split, but neither of them quite do what I want and there is no re.partition.

How should I partition strings on whitespace in Python with reasonable efficiency?

Answer 1

Try this:

s = "this is  a\nsentence"
re.split(r'(\W+)', s) # Notice parentheses and a plus sign.

Result would be:

['this', ' ', 'is', '  ', 'a', '\n', 'sentence']

Answer 2

Symbol of whitespace in re is '\s' not '\W'

Compare:

import re


s = "With a sign # written @ the beginning , that's  a\nsentence,"\
    '\nno more an instruction!,\tyou know ?? "Cases" & and surprises:'\
    "that will 'lways unknown **before**, in 81% of time$"


a = re.split('(\W+)', s)
print a
print len(a)
print

b = re.split('(\s+)', s)
print b
print len(b)

produces

['With', ' ', 'a', ' ', 'sign', ' # ', 'written', ' @ ', 'the', ' ', 'beginning', ' , ', 'that', "'", 's', '  ', 'a', '\n', 'sentence', ',\n', 'no', ' ', 'more', ' ', 'an', ' ', 'instruction', '!,\t', 'you', ' ', 'know', ' ?? "', 'Cases', '" & ', 'and', ' ', 'surprises', ':', 'that', ' ', 'will', " '", 'lways', ' ', 'unknown', ' **', 'before', '**, ', 'in', ' ', '81', '% ', 'of', ' ', 'time', '$', '']
57

['With', ' ', 'a', ' ', 'sign', ' ', '#', ' ', 'written', ' ', '@', ' ', 'the', ' ', 'beginning', ' ', ',', ' ', "that's", '  ', 'a', '\n', 'sentence,', '\n', 'no', ' ', 'more', ' ', 'an', ' ', 'instruction!,', '\t', 'you', ' ', 'know', ' ', '??', ' ', '"Cases"', ' ', '&', ' ', 'and', ' ', 'surprises:that', ' ', 'will', ' ', "'lways", ' ', 'unknown', ' ', '**before**,', ' ', 'in', ' ', '81%', ' ', 'of', ' ', 'time$']
61