Understanding Chapter 1 Example 2 from neuralnetworksanddeeplearning

Question

I am going through Chapter 1 of neuralnetworksanddeeplearning and didn't understand the second exercise (Sigmoid neurons simulating perceptrons, part II)

Show that in the limit as c→∞ the behaviour of this network of sigmoid neurons is exactly the same as the network of perceptrons. How can this fail when w⋅x+b=0 for one of the perceptrons?

I'm able to show that c→∞ behaves the same as network of perceptrons. But I'm not sure if I'm correct on the reason why w⋅x+b=0 would fail.

By substituting z = 0 for the sigmoid function (1 / (1 + e^-z), I get 1 / (1 + e^-0) which breaks down to 1 / (1 + 1) = 1/2

If the definition that 1/2 would trigger a 1 in the neuron, then I don't see why w⋅x+b=0 would fail.

Answer 1

You have more or less already answered your question. The transfer function of a perceptron is the step function H(z) which is zero for z<0 and 1 otherwise. The sigmoidal function S(c*z) for large c is is equivalent to the step function except at z=0 where H(z)=1 and S(c*z) = 0.5.

Answer 2

image source: Towards Data Science

As you can see in the above graph it is clear that for values < -5 and values > 5 the sigmoid function outputs a value close to 0 or 1 respectively. Here 5 is just used for representation but in reality this means that for extreme values of z the sigmoid function either outputs 0 or 1.

Therefore in your case, when you multiply the equation (w.x + b) with positive number c where c→∞ the value of z = c.(w.x + b) ~ -∞ (for w.x + b < 0) and z = c.(w.x + b) ~ ∞ (for w.x + b > 0). So, in these two cases you can expect a value close to 0 or 1. But when w.x + b = 0, z = 0 and sigmoid function outputs 0.5 or 1/2 which is not characteristic of a perceptron, which is described as "failing to mimic the behaviour of perceptron" in your question.