How to determine the number of layers and nodes of a neural network

Question

I am currently building a nn for a dataset with 387 features and 3000 samples. The outputs are 3 classes. I configured the network structure as following:

input->200->{300->100}->50->output

Did I choose the correct number of nodes and layers? How to determine the number of nodes of each layers(input,hidden and output)? Is there any rule?

Answer 1

Layers

As Yoshua Bengio, Head of Montreal Institute for Learning Algorithms remarks:

"Very simple. Just keep adding layers until the test error does not improve anymore."

A method recommended by Geoff Hinton is to add layers until you start to overfit your training set. Then you add dropout or another regularization method.

Nodes

For your task:

• Input layer should contain 387 nodes for each of the features.
• Output layer should contain 3 nodes for each class.
• Hidden layers I find gradually decreasing the number with neurons within each layer works quite well (this list of tips and tricks agrees with this when creating autoencoders for compression tasks). Perhaps try 200 in first hidden layer and 100 in the second; again it's a hyper-parameter to be optimised and is very dependent on dataset size.

Answer 2

Rules? As many as you want and none. Here is an excerpt from the Neural Network FAQ which is a good page to consult for basic questions:

7. A: How many hidden units should I use? ==========================================

   There is no way to determine a good network topology just from the number of inputs and outputs. It depends critically on the number of training examples and the complexity of the classification you are trying to learn. There are problems with one input and one output that require millions of hidden units, and problems with a million inputs and a million outputs that require only one hidden unit, or none at all.
   Some books and articles offer "rules of thumb" for choosing a topopology -- Ninputs plus Noutputs dividied by two, maybe with a square root in there somewhere -- but such rules are total garbage. Other rules relate to the number of examples available: Use at most so many hidden units that the number of weights in the network times 10 is smaller than the number of examples. Such rules are only concerned with overfitting and are unreliable as well.

In your case, however, one can definitely say that the network is much too complex (even if you applied strong regularization). Why so many hidden layers? Start with one hidden layer -- despite the deep learning euphoria -- and with a minimum of hidden nodes. Increase the hidden nodes number until you get a good performance. Only if not I would add further layers. Further, use cross validation and appropriate regularization.