Menu
Can I use batch normalization layer right after input layer and not normalize my data? May I expect to get similar effect/performance?
In keras functional it would be something like this:
x = Input (...)
x = Batchnorm(...)(x)
...
477
Not good for Recurrent Neural Networks Although batch normalization speeds-up training and generalization significantly in convolution neural networks, they are proven to be difficult to apply on recurrent architectures.
To answer your question: Yes, you should still standardize your inputs to a network that uses Batch Normalization.
Batch normalization solves a major problem called internal covariate shift. It helps by making the data flowing between intermediate layers of the neural network look, this means you can use a higher learning rate. It has a regularizing effect which means you can often remove dropout.
Batch Normalization vs Layer Normalization Batch normalization normalizes each feature independently across the mini-batch. Layer normalization normalizes each of the inputs in the batch independently across all features. As batch normalization is dependent on batch size, it's not effective for small batch sizes.
You can do it. But the nice thing about batchnorm, in addition to activation distribution stabilization, is that the mean and std deviation are likely migrate as the network learns.
Effectively, setting the batchnorm right after the input layer is a fancy data pre-processing step. It helps, sometimes a lot (e.g. in linear regression). But it's easier and more efficient to compute the mean and variance of the whole training sample once, than learn it per-batch. Note that batchnorm isn't free in terms of performance and you shouldn't abuse it.
169
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
