TensorFlow: tf.placeholder and tf.Variable - why is the dimension not required?

Question

I am learning TensorFlow from the example at: https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/2_BasicModels/linear_regression.ipynb

I have a few questions in the code below: When defining the placeholders and variables like X, Y, W, and b, why didn't we need to specify the dimension of them? How would the code allocate memory without knowing the size of these placeholders/variables? Thanks!

    # tf Graph Input
    X = tf.placeholder("float")
    Y = tf.placeholder("float")

    # Set model weights
    W = tf.Variable(rng.randn(), name="weight")
    b = tf.Variable(rng.randn(), name="bias")


    # Construct a linear model
    pred = tf.add(tf.mul(X, W), b)

Answer 1

TensorFlow's tf.placeholder() tensors do not require you to specify a shape, in order to allow you to feed tensors of different shapes in a later tf.Session.run() call. By default, a placeholder has a completely unconstrained shape, but you can constrain it by passing the optional shape argument. For example:

w = tf.placeholder(tf.float32)                      # Unconstrained shape
x = tf.placeholder(tf.float32, shape=[None, None])  # Matrix of unconstrained size
y = tf.placeholder(tf.float32, shape=[None, 32])    # Matrix with 32 columns
z = tf.placeholder(tf.float32, shape=[128, 32])     # 128x32-element matrix

When you create a placeholder, TensorFlow does not allocate any memory. Instead, when you feed the placeholder, in the call to tf.Session.run(), TensorFlow will allocate appropriately sized memory for the input (and subsequently for any necessary intermediate) tensors.

Note that tf.Variable objects typically do require a shape when you create them, and this shape is inferred from the first argument to the initializer. In your program, rng.randn() (an alias for numpy.random.randn()) returns a scalar value, and so the variables W and b will have scalar shape.

Although the placeholders in your code (X and Y) have unconstrained shape, some of the operators, such as tf.add() and tf.mul(), have additional requirements about the shape of their arguments (viz. that they are compatible with the NumPy broadcasting rules). Since TensorFlow doesn't know when you build the graph what the actual shapes of those tensors will be, it trusts that the user knows what they are doing, and performs the check dynamically (during the call to tf.Session.run()). If instead you constrain the shapes of the placeholders, you enable TensorFlow to perform some checks earlier, and doing this can help to reduce bugs.