Forming Bigrams of words in list of sentences with Python

Question

I have a list of sentences:

text = ['cant railway station','citadel hotel',' police stn'].  

I need to form bigram pairs and store them in a variable. The problem is that when I do that, I get a pair of sentences instead of words. Here is what I did:

text2 = [[word for word in line.split()] for line in text] bigrams = nltk.bigrams(text2) print(bigrams) 

which yields

[(['cant', 'railway', 'station'], ['citadel', 'hotel']), (['citadel', 'hotel'], ['police', 'stn']) 

Can't railway station and citadel hotel form one bigram. What I want is

[([cant],[railway]),([railway],[station]),([citadel,hotel]), and so on... 

The last word of the first sentence should not merge with the first word of second sentence. What should I do to make it work?

Answer 1

Using list comprehensions and zip:

>>> text = ["this is a sentence", "so is this one"] >>> bigrams = [b for l in text for b in zip(l.split(" ")[:-1], l.split(" ")[1:])] >>> print(bigrams) [('this', 'is'), ('is', 'a'), ('a', 'sentence'), ('so', 'is'), ('is', 'this'), ('this',      'one')] 

Answer 2

from nltk import word_tokenize  from nltk.util import ngrams   text = ['cant railway station', 'citadel hotel', 'police stn'] for line in text:     token = nltk.word_tokenize(line)     bigram = list(ngrams(token, 2))       # the '2' represents bigram...you can change it to get ngrams with different size