logistic / sigmoid function implementation numerical precision

Question

in scipy.special.expit, logistic function is implemented like the following:

if x < 0
    a = exp(x) 
    a / (1 + a) 
else 
    1 / (1 + exp(-x))

However, I have seen implementations in other languages/frameworks that simply do

1 / (1 + exp(-x))

I am wondering how much benefit the scipy version actually brings.

For very small x, the result approaches to 0. It works even if exp(-x) overflows to Inf.

Answer 1

It's really just for stability - putting in values that are very large in magnitude might return unexpected results otherwise.

If expit was implemented just as 1 / (1 + exp(-x)) then putting a value of -710 into the function would return nan, whereas -709 would give a value close to zero as expected. This is because exp(710) is too big to be a double.

The branching in the code just means that this scenario is avoided.

See also this question and answer on Stack Overflow.