Tensorflow: When use tf.expand_dims?

Question

Tensorflow tutorials include the use of tf.expand_dims to add a "batch dimension" to a tensor. I have read the docs for this function but it still is rather mysterious to me. Does anyone know exactly under what circumstances this must be used?

My code is below. My intent is to calculate a loss based on the distance between the predicted and actual bins. (E.g. if predictedBin = 10 and truthBin = 7 then binDistanceLoss = 3).

batch_size = tf.size(truthValues_placeholder) labels = tf.expand_dims(truthValues_placeholder, 1) predictedBin = tf.argmax(logits) binDistanceLoss = tf.abs(tf.sub(labels, logits)) 

In this case, do I need to apply tf.expand_dims to predictedBin and binDistanceLoss? Thanks in advance.

Answer 1

expand_dims will not add or reduce elements in a tensor, it just changes the shape by adding 1 to dimensions. For example, a vector with 10 elements could be treated as a 10x1 matrix.

The situation I have met to use expand_dims is when I tried to build a ConvNet to classify grayscale images. The grayscale images will be loaded as matrix of size [320, 320]. However, tf.nn.conv2d require input to be [batch, in_height, in_width, in_channels], where the in_channels dimension is missing in my data which in this case should be 1. So I used expand_dims to add one more dimension.

In your case, I do not think you need expand_dims.

Answer 2

To add to Da Tong's answer, you may want to expand more than one dimension at the same time. For instance, if you are performing TensorFlow's conv1d operation on vectors of rank 1, you need to feed them with rank three.

Performing expand_dims several times is readable, but might introduce some overhead into the computational graph. You can get the same functionality in a one-liner with reshape:

import tensorflow as tf  # having some tensor of rank 1, it could be an audio signal, a word vector... tensor = tf.ones(100) print(tensor.get_shape()) # => (100,)  # expand its dimensionality to fit into conv2d tensor_expand = tf.expand_dims(tensor, 0) tensor_expand = tf.expand_dims(tensor_expand, 0) tensor_expand = tf.expand_dims(tensor_expand, -1) print(tensor_expand.get_shape()) # => (1, 1, 100, 1)  # do the same in one line with reshape tensor_reshape = tf.reshape(tensor, [1, 1, tensor.get_shape().as_list()[0],1]) print(tensor_reshape.get_shape()) # => (1, 1, 100, 1) 

NOTE: In case you get the error TypeError: Failed to convert object of type <type 'list'> to Tensor., try to pass tf.shape(x)[0] instead of x.get_shape()[0] as suggested here.

Hope it helps!
Cheers,
Andres