Using logarithms to normalize a vector to avoid overflow

Question

Problem with arithmetic using logarithms to avoid numerical underflow (take 2)

Having seen the above and having seen softmax normalization I was trying to normalize a vector while avoiding overflow -

that is if I have an array x[1], x[2] x[3], x[4], ... , x[n]

the normalized form for me has the sum of squares of elements as 1.0 and is obtained by dividing each element by sqrt(x[1]*x[1]+x[2]*x[2]+...+x[n]*x[n])

now the sum of squares can overflow even if the square root is small enough to fit into a floating point variable, so I imagined one could do something like s=(2*log(fabs(x[1]))+2*log(fabs(x[2]))+...+2*log(fabs(x[n])))/2

and calculating the elements as

exp(log(fabs(x[1]))-s), ..., exp(log(fabs(x[n]))-s

BUT

The above is incorrect as log(A+B) is not log(A)+log(B) - now is there a way to do vector normalization that avoids overflow better?

Answer 1

Instead of

norm  = sqrt(x[1] * x[1] + ... + x[n] * x[n])

you might want to divide the elements of the vector by the maximum possible value before squaring

max_x = max(x[1], ..., x[n])
y[1] = x[1] / max_x / n
...
y[n] = x[n] / max_x / n
norm = n * sqrt(y[1] * y[1] + ... + y[n] * y[n]) * max_x

The norm of the y vector should then be equal or smaller than zero. The value of n * max_x could still overflow, so you need to be careful there, too, that the operations are executed in a non-overflowing order.

Answer 2

You seem to be making the assumption that:

log(x^2 + y^2)

is the same as:

log(x^2) + log(y^2)

However, this isn't correct, as you can't simplify the logarithm of a sum like that.