Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Same math logic, different results?

How am I getting different results from the following two queries? a,b,c are float and I would assume these return the same results but they are slightly off.

SELECT (a-b)+(c) 
FROM
    (
            select  sum([Actual Freight Per Line Amt]) a, 
                    sum([fedex charge per line amt]) b, 
                    sum([inbound freight cost]) c 
             from stg.invoices where year([gl date]) = '2016'
    ) foo

results in: -5822899.31314175

&

SELECT SUM((a-b)+(c))
FROM
    (
            select  [Actual Freight Per Line Amt] a, 
                    [fedex charge per line amt] b, 
                    [inbound freight cost] c 
             from stg.invoices where year([gl date]) = '2016'
    ) foo

results in: -5796251.59304654

like image 560
precose Avatar asked Feb 06 '23 04:02

precose


1 Answers

When you are doing arithmetic on a lot of floating point numbers, the ordering matters. A canonical example is:

1,000,000,000,000,000,000,000,000,000 + -1,000,000,000,000,000,000,000,000,000 + 38

If this is evaluated as:

(1,000,000,000,000,000,000,000,000,000 + -1,000,000,000,000,000,000,000,000,000) + 38

You'll get 38. The "38" is so much smaller than the other number that a floating point representation cannot represent the value.

If it is evaluated at:

1,000,000,000,000,000,000,000,000,000 + (-1,000,000,000,000,000,000,000,000,000 + 38)

You'll get 0.

I recommend that you use decimals for the calculation.

like image 148
Gordon Linoff Avatar answered Feb 07 '23 19:02

Gordon Linoff