Things to try when Neural Network not Converging

Question

One of the most popular questions regarding Neural Networks seem to be:

Help!! My Neural Network is not converging!!

See here, here, here, here and here.

So after eliminating any error in implementation of the network, What are the most common things one should try??

I know that the things to try would vary widely depending on network architecture. But tweaking which parameters (learning rate, momentum, initial weights, etc) and implementing what new features (windowed momentum?) were you able to overcome some similar problems while building your own neural net?

Please give answers which are language agnostic if possible. This question is intended to give some pointers to people stuck with neural nets which are not converging..

Answer 1

If you are using ReLU activations, you may have a "dying ReLU" problem. In short, under certain conditions, any neuron with a ReLU activation can be subject to a (bias) adjustment that leads to it never being activated ever again. It can be fixed with a "Leaky ReLU" activation, well explained in that article.

For example, I produced a simple MLP (3-layer) network with ReLU output which failed. I provided data it could not possibly fail on, and it still failed. I turned the learning rate way down, and it failed more slowly. It always converged to predicting each class with equal probability. It was all fixed by using a Leaky ReLU instead of standard ReLU.

Answer 2

If we are talking about classification tasks, then you should shuffle examples before training your net. I mean, don't feed your net with thousands examples of class #1, after thousands examples of class #2, etc... If you do that, your net most probably wouldn't converge, but would tend to predict last trained class.