TensorFlow LSTM Generative Model

Question

I'm working off the LSTM language model tutorial discussed here.

With language models, it's common to use the model to generate a new sentence from scratch after training (i.e. sample from the model).

I'm new to TensorFlow but I'm trying to use my trained model to generate new words until the end-of-sentence marker.

My initial attempt:

x = tf.zeros_like(m.input_data)
state = m.initial_state.eval()
for step in xrange(m.num_steps):
    state = session.run(m.final_state,
                               {m.input_data: x,
                                m.initial_state: state})
    x = state

It fails with error:

ValueError: setting an array element with a sequence.

Answer 1

The issue here seems to be the m.input_data: x mapping in the feed_dict passed session.run(). In this case, TensorFlow expects that x is a numpy array (or some object that can be implicitly converted to a numpy array), but the value is a TensorFlow Tensor (the result of tf.zeros_like()).

Fortunately, the solution is simple. Replace x = tf.zeros_like(m.input_data) with the following:

x = tf.zeros_like(m.input_data).eval()

...which ensures that x is converted to a numpy array.

(Note that a more direct way to achieve this would be to construct the initial x as a numpy array of the appropriate size.)