tokenize while preserving newline and paragraph structure

Question

Let's say I have a text that I want to study. The number of sentences and paragraphs are important to be preserved as they are (dots trigger end of sentence, newline trigger start of a new paragraph). Let's say I need to first tokenize my text:

>>> from nltk import word_tokenize as tokenize
>>> tokenize('How\'s life? Aren\'t you feeling good\n bro?')
['How', "'s", 'life', '?', 'Are', "n't", 'you', 'feeling', 'good', 'bro', '?']

As you see, the output is a list that disregards any information beyond words. I could try to do something like:

>>> s = ['How', "'s", 'life', '?', 'Are', "n't", 'you', 'feeling', 'good', 'bro', '?']
>>> " ".join(s)
"How 's life ? Are n't you feeling good bro ?"

but it's not enough because paragraph structure is already lost. Is there an easy way to do this using NLTK's tokenizer? I'd rather not resort to a different library as the first solution.

Answer 1

You probably want to change tokenizers. There are several different ones included with nltk. If you want to retain the significance of newlines then maybe try one of the line oriented tokenizers, or consider splitting the string on newlines yourself and feeding in the parts one at a time. This should enable you to mix the newlines back into your data somehow. (For example, you might just keep track of the len() of the tokenized output at each newline.)