Tensorflow flatten vs numpy flatten function effect on machine learning training

Question

I am starting with deep learning stuff using keras and tensorflow. At very first stage i am stuck with a doubt. when I use tf.contrib.layers.flatten (Api 1.8) for flattening a image (could be multichannel as well).

How is this different than using flatten function from numpy? How does this affect the training. I can see the tf.contrib.layers.flatten is taking longer time than numpy flatten. Is it doing something more?

This is a very close question but here the accepted answer includes Theano and does not solve my doubts exactly.

Example: Lets say i have a training data of (10000,2,96,96) shape. Now I need the output to be in (10000,18432) shape. I can do this using tensorflow flatten or by using numpy flatten like

X_reshaped = X_train.reshape(*X_train.shape[:1], -2)

what difference does it make in training and which is the best practice?

Answer 1

The biggest difference between np.flatten and tf.layers.flatten (or tf.contrib.layers.flatten) is that numpy operations are applicable only to static nd arrays, while tensorflow operations can work with dynamic tensors. Dynamic in this case means that the exact shape will be known only at runtime (either training or testing).

So my recommendation is pretty simple:

• If the input data is static numpy array, e.g. in pre-processing, use np.flatten. This avoids unnecessary overhead and returns numpy array as well.
• If the data is already a tensor, use any of the flatten ops provided by tensorflow. Between those, tf.layers.flatten is better choice since tf.layers API is more stable than tf.contrib.*.

Answer 2

• Use numpy directly on your data, without participation of a neural network. This is for preprocessing and postprocessing only
• Use TF or Keras layers inside models if this operation is needed for some reason in the model. This will assure model connectivity and proper backpropagation

Models are symbolic graphs meant to create Neural Networks that can be trained. There will be a proper connection and backpropagation will work properly when you have a graph connected from input to output.

If you don't intend to create a network, don't use a TF layer. If your goal just to flatten an array, you don't need a neural network.

Now if inside a model you need to change the format of the data without losing connection and backpropagation, then go for the flatten layer.

Answer 3

The flatten function in numpy does a complete array flattening, meaning that you end up with a single axis of data (1 dimension only). For example,

import numpy as np
a = np.arange(20).reshape((5,4))
print(a)

print(a.flatten().shape)

In the previous example, you end up with a 1-d array of 20 elements. In tensorflow, the flatten layer (tf.layers.flatten) preserves the batch axis (axis 0). In the previous example, with tensorflow, you would still have a shape of (5,4).

In any case, there is no effect on training if you use flatten in an equivalent way. However, you should avoid using numpy when working with tensorflow, since almost all numpy operations have their tensorflow counterparts. Tensorflow and numpy rely on different runtime libraries and combining both could be runtime inefficient.

Moreover, avoid using contrib package layers, when they already exist in the main package (use tf.layers.flatten instead of tf.contrib.layers.flatten).

For a more general performance comparison between numpy and tensorflow, have a look at this question: Tensorflow vs. Numpy Performance

Answer 4

Difference

When you use tensorflow flatten, it gets added as an operation (op) in the graph. It can operate only on tensors. Numpy on the other hand works on actual numpy arrays. The usage is completely different.

Usage

You would use tensorflow op if this is an operation in the training process such as resizing before feeding to the next layer.

You would use numpy op when you want to operate on actual value at that time, like reshaping for calculating accuracy at the end of training step.

So if you had a task of

tensor A -> reshape -> matrix_mul

If you use tensorflow for reshape, you can directly run the matrix_mul from session.

If you use numpy however, you'd have to run the operation in two stages (two session calls).

1. You calculate tensor A

2. You reshape it in numpy.

3. Run the matrix_mul by "feeding" in reshaped array.

Performance

I haven't benchmarked anything but I'd say for just a reshape operation as standalone, numpy would be faster (ignoring gpu ) , but in a process where reshape is an intermediate op, tensorflow should be faster.