Neural networks and the XOR function

Question

I'm playing with a neural network I implemented myself: it's a trivial forward network using RPROP as a learning algorithm as the only "plus" compared to the basic design.

The network scores decently when I test it against MNIST or when I attempt at image compression, but when I try to model something as simple as the XOR function, sometimes during learning it gets trapped into a local minima, and outputs the following truth table:

0 XOR 0 = 1.4598413968251171e-171
1 XOR 0 = 0.9999999999999998
0 XOR 1 = 0.9999999999999998
1 XOR 1 = 0.5

Often the result after the training is correct, but sometimes 1 XOR 1 outputs 0.5 instead of 1 as it should. It does not really always happens with XOR(1,1), but with other inputs as well. Being the XOR function a "classical" in the literature of back propagation I wonder what's happening here, especially given that my network appears to learn more complex (but perhaps less non-linear) tasks just fine.

My wild guess is that's something wrong with the biases.

Any hint?

Note 1: the network layout above is 2|3|1, but does not change much when I use more hidden units, certain learning attempts still go wrong.

Note 2: I put the implementation into a Gist: https://gist.github.com/antirez/e45939b918868b91ec6fea1d1938db0d

Answer 1

The problem was due to a bug in my implementation: the bias unit of the NN immediately before the output unit was not computed correctly. After fixing the code the XOR function is always computed right.