Where Dropout should be inserted.? Fully Connected Layer.? Convolutional Layer.? or Both.? [closed]

Question

I would like to get your feedback on where Dropout should be inserted?

Should it be located in the Fully connected layer (Dense), or Convolutional Layer.? or Both.?

Thank you for your feedback in advance.

Answer 1

Usually, dropout is placed on the fully connected layers only because they are the one with the greater number of parameters and thus they're likely to excessively co-adapting themselves causing overfitting.

However, since it's a stochastic regularization technique, you can really place it everywhere. Usually, it's placed on the layers with a great number of parameters, but no one denies you from applying it to convolutional layer instead (that got a lower number of parameters respect to the fc layers).

Moreover, the drop probability should be changed according to the impact of the regularization you want.

A rule of thumb is to set the keep probability (1 - drop probability) to 0.5 when dropout is applied to fully connected layers whilst setting it to a greater number (0.8, 0.9, usually) when applied to convolutional layers.

Just a note: since in every machine learning framework dropout is implemented in its "inverted" version, you should have to lower your learning rate in order to overcome the "boost" that the dropout probability gives to the learning rate. For a more comprehensive assessment about it: https://pgaleone.eu/deep-learning/regularization/2017/01/10/anaysis-of-dropout/

Answer 2

Dropout is just a regularization technique for preventing overfitting in the network. It sets a node's weight to zero with a given probability during training, reducing the number of weights required for training at each iteration. It can be applied for each layer of the network (regardless if it is fully connected or convolutional), or after selected layers. To which layers dropout is applied is really just a design decision for what results in best performance.