tf.Variable with dynamic shape from input placeholder

Question

I am trying to build a network, where my requirement is hidden unit shape should be according to input shape, so that user can give any length of input. What I am trying to do is :

import tensorflow as tf

n_hidden_1=100
input_ = tf.placeholder(name='input_data',shape=[None,None],dtype=tf.float32)
variable_data = tf.Variable(tf.random_normal([tf.shape(input_)[1], n_hidden_1]))

bias_ = tf.Variable(tf.random_normal([n_hidden_1]))

output = tf.matmul(input_,variable_data)+bias_

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(output,feed_dict={input_:[[1,2,3,4,5]]}))

I want

variable_data = tf.Variable(tf.random_normal([tf.shape(input_)[1], n_hidden_1]))

shape from input placeholder, but it's giving an error. That's why I tried this approach:

variable_data = tf.Variable(tf.random_normal([tf.shape(input_)[1], n_hidden_1]),validate_shape=False)

But then I am getting another error:

InvalidArgumentError: You must feed a value for placeholder tensor 'input_data_7' with dtype float and shape [?,?]
     [[Node: input_data_7 = Placeholder[dtype=DT_FLOAT, shape=[?,?], _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

How can I fetch shape int from placeholder for variable shape.

Answer 1

A tf.Variable cannot really have a dynamic shape, because it does not make sense for their purpose. In tensorflow, a variable is meant to be a model parameter that is optimized, typically by SGD. During optimization you typically assume that your cost function and the space it is defined on does not vary at each step. Not doing so — especially in a random, non-scheduled way — would result in an ill-defined optimization problem.

However, what you can do is having a variable where we defer the definition of its (static) shape at runtime, when the first sample comes in. You were almost there: you simply have to provide the sample to global_variables_initializer, which will initialize the variables to the correct shape and value:

import tensorflow as tf


n_hidden_1=100
input_ = tf.placeholder(name='input_data',shape=[None,None],dtype=tf.float32)
variable_data = tf.Variable(tf.random_normal([tf.shape(input_)[1], n_hidden_1]), validate_shape=False)

bias_ = tf.Variable(tf.random_normal([n_hidden_1]))

output = tf.matmul(input_,variable_data)+bias_

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer(), feed_dict={input_:[[1,2,3,4,5]]})
    print(sess.run(output, feed_dict={input_:[[1,2,3,4,5]]}))