Neural network weights explode in linear unit

Question

I am currently implementing a simple neural network and the backprop algorithm in Python with numpy. I have already tested my backprop method using central differences and the resulting gradient is equal.

However, the network fails to approximate a simple sine curve. The network hast one hidden layer (100 neurons) with tanh activation functions and a output layer with a linear activation function. Each unit hast also a bias input. The training is done by simple gradient descent with a learning rate of 0.2.

The problem arises from the gradient, which gets with every epoch larger, but I don't know why? Further, the problem is unchanged, if I decrease the learning rate.

EDIT: I have uploaded the code to pastebin: http://pastebin.com/R7tviZUJ

Answer 1

There are two things you can try, maybe in combination:

1. Use a smaller learning rate. If it is too high, you may be overshooting the minimum in the current direction by a lot, and so your weights will keep getting larger.

2. Use smaller initial weights. This is related to the first item. A smaller learning rate would fix this as well.