Menu
Is the batchnorm momentum convention (default=0.1) correct as in other libraries e.g. Tensorflow it seems to usually be 0.9 or 0.99 by default? Or maybe we are just using a different convention?
538
Pytorch batch normalization is a process of training the neural network. During training the network this layer keep guessing its computed mean and variance. Code: In the following code, we will import some libraries from which we can train the neural network and also evaluate its computed mean and variance.
Momentum is the “lag” in learning mean and variance, so that noise due to mini-batch can be ignored. Actual(light) and lagged(bold) values with momentum 0.99 and 0.75. By default, momentum would be set a high value about 0.99, meaning high lag and slow learning. When batch sizes are small, the no.
BatchNorm2d is the number of dimensions/channels that output from the last layer and come in to the batch norm layer.
It seems that the parametrization convention is different in pytorch than in tensorflow, so that 0.1 in pytorch is equivalent to 0.9 in tensorflow.
To be more precise:
In Tensorflow:
running_mean = decay*running_mean + (1-decay)*new_value
In PyTorch:
running_mean = (1-decay)*running_mean + decay*new_value
This means that a value of decay in PyTorch is equivalent to a value of (1-decay) in Tensorflow.
151
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
