Is there any difference between TrainingHelper and GreedyEmbeddingHelper in Tensorflow r.1.1(predict result different)?

Question

I'm a beginner in tensorflow. I want to study tensorflow by using this tutorial.

After reading this tutorial, I want to run this code by using my data(Korea title for tokenizing) In training model(use TrainingHelper), the prediction results seems to be OK. But In inference model(use GreedyEmbeddingHelper), the prediction results are really bad(even though using train data). It looks like first epoch's training model prediction. Is there any difference TrainingHelper and GreedyEmbeddingHelper?

I think the difference between tutorial and my code is just hyper-parameter.

Answer 1

TrainingHelper is for use at training time, when (one of the) inputs to your decoder RNN is the ground truth from the previous time step. Because the ground truth is not available at inference time, you instead feed in the decoder output from the previous time step.

For example, consider the target sentence "I like pizza". At training time, when decoding the word "pizza", the decoding RNN will receive the following inputs:

1. The ground truth from the previous time step, e.g. the embedding for the word "like" (using the target embedding).
2. The context from the previous time step.
3. The hidden state from the previous time step.

At inference time, the decoding RNN will still receive 2 and 3. However, instead of the ground truth, it will take the decoder output from the previous time step (a one-hot encoding equal to the length of the target vocabulary, e.g. the word your decoder guessed at the previous time step), run it through the target embedding, and use that as an input instead.