Can I use Layer Normalization with CNN?

Question

I see the Layer Normalization is the modern normalization method than Batch Normalization, and it is very simple to coding in Tensorflow. But I think the layer normalization is designed for RNN, and the batch normalization for CNN. Can I use the layer normalization with CNN that process image classification task? What are the criteria for choosing batch normalization or layer?

Answer 1

You can use Layer normalisation in CNNs, but i don't think it more 'modern' than Batch Norm. They both normalise differently. Layer norm normalises all the activations of a single layer from a batch by collecting statistics from every unit within the layer, while batch norm normalises the whole batch for every single activation, where the statistics is collected for every single unit across the batch.

Batch norm is generally preferred over layer norm as it tries to normalise every activation to a unit gaussian distribution, while layer norm tries to get the 'average' of all activations to unit gaussian. But if the batch size is too small to collect reasonable statistics, then layer norm is preferred.