Predicting Missing Word in Sentence

Question

How can I predict a word that's missing from a sentence?

I've seen many papers on predicting the next word in a sentence using an n-grams language model with frequency distributions from a set of training data. But instead I want to predict a missing word that's not necessarily at the end of the sentence. For example:

I took my ___ for a walk.

I can't seem to find any algorithms that take advantage of the words after the blank; I guess I could ignore them, but they must add some value. And of course, a bi/trigram model doesn't work for predicting the first two words.

What algorithm/pattern should I use? Or is there no advantage to using the words after the blank?

Answer 1

Tensorflow has a tutorial to do this: https://www.tensorflow.org/versions/r0.9/tutorials/word2vec/index.html

Incidentally it does a bit more and generates word embeddings, but to get there they train a model to predict the (next/missing) word. They also show using only the previous words, but you can apply the same ideas and add the words that follow.

They also have a bunch of suggestions on how to improve the precision (skip ngrams).

Somewhere at the bottom of the tutorial you have links to working source-code.

The only thing to be worried about is to have sufficient training data.

Answer 2

So, when I've worked with bigrams/trigrams, an example query generally looked something like "Predict the missing word in 'Would you ____'". I'd then go through my training data and gather all the sets of three words matching that pattern, and count the things in the blanks. So, if my training data looked like:

would you not do that
would you kindly pull that lever
would you kindly push that button
could you kindly pull that lever

I would get two counts for "kindly" and one for "not", and I'd predict "kindly". All you have to do for your problem is consider the blank in a different place: "____ you kindly" would get two counts for "would" and one for "could", so you'd predict "would". As far as the computer is concerned, there's nothing special about the word order - you can describe whatever pattern you want, from your training data. Does that make sense?