Overflow / math range error for log or exp

Question

Line of code in question:

summing +=  yval * np.log(             sigmoid(np.dot(w.transpose(),xi.transpose()))) 
        +(1-yval)* np.log(max(0.001, 1-sigmoid(np.dot(w.transpose(),xi.transpose()))))

Error:

File "classify.py", line 67, in sigmoid
return 1/(1+ math.exp(-gamma))
OverflowError: math range error

The sigmoid function is just 1/(1+ math.exp(-gamma)).

I'm getting a math range error. Does anyone see why?

Answer 1

You can avoid this problem by using different cases for positive and negative gamma:

def sigmoid(gamma):
  if gamma < 0:
    return 1 - 1/(1 + math.exp(gamma))
  else:
    return 1/(1 + math.exp(-gamma))

The math range error is likely because your gamma argument is a large negative value, so you are calling exp() with a large positive value. It is very easy to exceed your floating point range that way.

Answer 2

The problem is that, when gamma becomes large, math.exp(gamma) overflows. You can avoid this problem by noticing that

sigmoid(x) = 1 / (1 + exp(-x))
           = exp(x) / (exp(x) + 1)
           = 1 - 1 / (1 + exp(x))
           = 1 - sigmoid(-x)

This gives you a numerically stable implementation of sigmoid which guarantees you never even call math.exp with a positive value:

def sigmoid(gamma):
    if gamma < 0:
        return 1 - 1 / (1 + math.exp(gamma))
    return 1 / (1 + math.exp(-gamma))