Numerically Stable Implementation

Question

I need to compute a normalized exponential of a vector in Matlab.

Simply writing

res = exp(V)/sum(exp(V))

overflows in an element of V is greater than log(realmax) = 709.7827. (I am not sure about underflow conditions.)

How should I implement it to avoid numerical instability?

Update: I received excellent responses about how to avoid overflow. However, I am still happy to hear your thoughts about the possibility of underflow in the code.

Answer 1

The following approach avoids the overflow by subtracting the exponents and then taking the exponential, instead of dividing the exponentials:

res = 1./sum(exp(bsxfun(@minus, V(:), V(:).')))

As a general rule, overflow can be avoided by working in the log domain for as long as possible, and taking the exponential only at the end.