search similar meaning phrases with nltk

Question

I have a bunch of unrelated paragraphs, and I need to traverse them to find similar occurrences such as that, given a search where I look for object falls, I find a boolean True for text containing:

• Box fell from shelf
• Bulb shattered on the ground
• A piece of plaster fell from the ceiling

And False for:

• The blame fell on Sarah
• The temperature fell abruptly

I am able to use nltk to tokenise, tag and get Wordnet synsets, but I am finding it hard to figure out how to fit nltk's moving parts together to achieve the desired result. Should I chunk before looking for synsets? Should I write a context-free grammar? Is there a best practice when translating from treebank tags to Wordnet grammar tags? None of this is explained in the nltk book, and I couldn't find it on the nltk cookbook yet.

Bonus points for answers that include pandas in the answer.

---

[ EDIT ]:

Some code to get things started

In [1]:

from nltk.tag import pos_tag
from nltk.tokenize import word_tokenize
from pandas import Series

def tag(x):
    return pos_tag(word_tokenize(x))

phrases = ['Box fell from shelf',
           'Bulb shattered on the ground',
           'A piece of plaster fell from the ceiling',
           'The blame fell on Sarah',
           'Berlin fell on May',
           'The temperature fell abruptly']

ser = Series(phrases)
ser.map(tag)

Out[1]:

0    [(Box, NNP), (fell, VBD), (from, IN), (shelf, ...
1    [(Bulb, NNP), (shattered, VBD), (on, IN), (the...
2    [(A, DT), (piece, NN), (of, IN), (plaster, NN)...
3    [(The, DT), (blame, NN), (fell, VBD), (on, IN)...
4    [(Berlin, NNP), (fell, VBD), (on, IN), (May, N...
5    [(The, DT), (temperature, NN), (fell, VBD), (a...
dtype: object

Answer 1

The way I would do it is the following:

Use nltk to find nouns followed by one or two verbs. In order to match your exact specifications I would use Wordnet: The only nouns (NN, NNP, PRP, NNS) that should be found are the ones that are in a semantic relation with "physical" or "material" and the only verbs (VB, VBZ, VBD, etc...) that should be found are the ones that are in a semantic relation with "fall".

I mentioned "one or two verbs" because a verb can be preceded by an auxiliary. What you could also do is create a dependency tree to spot subject-verb relations, but it does not seem to be necessary in this case.

You might also want to make sure you exclude location names and keep person names (Because you would accept "John has fallen" but not "Berlin has fallen"). This can also be done with Wordnet, locations have the tag 'noun.location'.

I am not sure in which context you would have to convert the tags so I cannot provide a proper answer to that, in seems to me that you might not need that in this case: You use the POS tags to identify nouns and verbs and then you check if each noun and verb belong to a synset.

Hope this helps.