Neural Network with input of 0

Question

Imagine you have a Neural Network with Sigmoids. It has an input x and so a node would output tanh⁡x to a connection. The connection would then output w∗tanhx where w is the weight of the connection.

The problem is, what if an input is 0 and the desire output should be something like 1? Well if the input is 0, the output of the connection would be w∗tanh0=0. So if the input is 0, then then output will always be 0 no matter how many nodes or connections you add or how much the weights on the connections are.

How would you make a simple network where an input of 0 will give you something other than 0?

Answer 1

Aside

One thing to notice is that the function for each layer should be of the form:

z = transpose(W) * X output = tanh(z)

where you are multiplying by the weight vector before passing into the transfer function (tanh). This however does not answer your question or fix your problem.

First way

To answer your question, you should add a bias unit to your input. One such way of doing this is to use a constant input of 1 as an input.

So if you have pseudo code: number_of_inputs = 10 then you'd instead use pseudo code: number_of_inputs = 11; input[10] = 1.

This allows your output function to be of the same form as above.

Second Way

You can also add a bias unit outside of your transfer function. That is, your output could be of the form:

z = transpose(W) * X output = tanh(z) + b

The update rule for b then would be based on the derivative of your loss, but not the derivative of your transfer.

Answer 2

You should add a pseudo input whose value is constant 1. This input can then be used as a weighted bias for neurons in the first layer.