Menu
I came across this equation in Alex's paper on local response normalization:
As we see above, the power is raised after we calculate the sum, multiply it with alpha and then add once k is added.
However, I see in TensorFlow documentation it is showed as
sqr_sum[a, b, c, d] =
sum(input[a, b, c, d - depth_radius : d + depth_radius + 1] ** 2)
output = input / (bias + alpha * sqr_sum ** beta)
where the beta is raised only for the sum.
Why is there a discrepancy here?
Also, when I looked into the TensorFlow code itself, I saw this:
output[b, r, c, d] /= (
np.power(bias + alpha * np.sum(patch * patch), beta))
which looks correct?
I am kind of confused here. Can someone correct me please?
247
Local Response Normalization (LRN) was first introduced in AlexNet architecture where the activation function used was ReLU as opposed to the more common tanh and sigmoid at that time. Apart from the reason mentioned above, the reason for using LRN was to encourage lateral inhibition.
Local Response Normalization (LRN) is not used by VGG since it increases memory usage and training time without improving accuracy. VGG contains three completely connected layers, the first two of which each has 4096 channels and the third of which has 1000 channels, one for each class.
Local Response meanss a hazardous materials emergency response in the local governmental area where team members normally conduct emergency response activities and those areas where the local government has a hazardous materials mutual response agreement in place and the responding team does not respond as a state team ...
This was a bug in the Tensorflow documentation.
It was fixed by this commit on Feb 26: https://github.com/tensorflow/tensorflow/commit/ab48dbd4ac2095548a5bc8505e08e751d409727f#diff-632987400e5affbcdba4533444460b0e
100
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
