Where to add dropout in neural network?

Question

I have seen description about the dropout in different parts of the neural network:

1. dropout in the weight matrix,

2. dropout in the hidden layer after the matrix multiplication and before relu,

3. dropout in the hidden layer after the relu,

4. and dropout in the output score prior to the softmax function

I am a little confused about where I should perform the dropout. Could someone help elaborate about that? Thanks!

Answer 1

So:

1. The first usage you've described is called a dropout on weights.
2. The second and the third usage you've described are the same and they're commonly described as dropout on activations. One may easily notice that it might be represented in terms of dropout on weights when a whole row (or column - depending on implementation) is turned off.
3. In 4th case it's not a proper usage of dropout - the layer you want to use dropout on is an output layer - so it's not the best idea to use dropout there.