Training a neural network to add

Question

I need to train a network to multiply or add 2 inputs, but it doesn't seem to approximate well for all points after 20000 iterations. More specifically, I train it on the whole dataset and it approximates well for the last points, but it seems like it isn't getting any better for the first endpoints. I normalize the data so that it is between -0.8 and 0.8. The network itself consists of 2 inputs 3 hidden neurons and 1 output neuron. I also set the network's learning rate to 0.25, and use as a learning function tanh(x).

It approximates really well for points that are trained last in the dataset, but for the first points it seems like it can't approximate well. I wonder what it is, that isn't helping it adjust well, whether it is the topology I am using, or something else?

Also how many neurons are appropriate in the hidden layer for this network?

Answer 1

A network consisting of a single neuron with weights={1,1}, bias=0 and linear activation function performs the addition of the two input numbers.

Multiplication may be harder. Here are two approaches that a net can use:

1. Convert one of the numbers to digits (for example, binary) and perform multiplication as you did in elementary school. a*b = a*(b0*2^0 + b1*2^1 + ... + bk*2^k) = a*b0*2^0 + a*b1*2^1 + ... + a*bk*2^k. This approach is simple, but requires variable number of neurons proportional to the length (logarithm) of the input b.
2. Take logarithms of the inputs, add them and exponentiate the result. a*b = exp(ln(a) + ln(b)) This network can work on numbers of any length as long as it can approximate the logarithm and exponent well enough.

Answer 2

It may be too late, but a simple solution is to use a RNN (Recurrent Neural Network).

After converting your numbers to digits, your NN will take a couple of digits from the sequence of digits from left to right.

The RNN has to loop one of its output so that it can automatically understand that there is a digit to carry (if the sum is 2, write a 0 and carry 1).

To train it, you'll need to give it the inputs consisting of two digits (one from the first number, the second from the second number) and the desired output. And the RNN will end up finding how to do the sum.

Notice that this RNN will only need to know the 8 following cases to learn how to sum two numbers:

• 1 + 1, 0 + 0, 1 + 0, 0 + 1 with carry
• 1 + 1, 0 + 0, 1 + 0, 0 + 1 without carry