How to perform mean subtraction and normalization with Tensorflow

Question

On http://cs231n.github.io/neural-networks-2/ it is mentioned that for convolutional neural networks it is preferred to preprocess data using mean subtraction and normalization techniques.

I was just wondering how would it be best approached using Tensorflow.

Mean substraction

X -= np.mean(X)

Normalization

X /= np.std(X, axis = 0)

Answer 1

You're looking for

1. tf.image.per_image_whitening(image) if you use Tensorflow version < r0.12
2. tf.image.per_image_standardization(image) otherwise.

Linearly scales image to have zero mean and unit norm.

This op computes (x - mean) / adjusted_stddev, where mean is the average of all values in image, and adjusted_stddev = max(stddev, 1.0/sqrt(image.NumElements())).

Answer 2

Looking in the source code for feature columns I noticed for real_valued_column types there is a keyword argument normalizer that can take as an argument a default normalization function to apply to each element of the tensor:

real_valued_column("col_name", normalizer = lambda x: (x-X.mean())/X.std())

Where X is your data. I think the advantage here is the normalization can be applied in the course of a tensor flow graph on a purpose built machine. Also the normalization function can be easily customized.