How am I getting different results from the following two queries? a,b,c are float and I would assume these return the same results but they are slightly off.
SELECT (a-b)+(c)
FROM
(
select sum([Actual Freight Per Line Amt]) a,
sum([fedex charge per line amt]) b,
sum([inbound freight cost]) c
from stg.invoices where year([gl date]) = '2016'
) foo
results in: -5822899.31314175
&
SELECT SUM((a-b)+(c))
FROM
(
select [Actual Freight Per Line Amt] a,
[fedex charge per line amt] b,
[inbound freight cost] c
from stg.invoices where year([gl date]) = '2016'
) foo
results in: -5796251.59304654
When you are doing arithmetic on a lot of floating point numbers, the ordering matters. A canonical example is:
1,000,000,000,000,000,000,000,000,000 + -1,000,000,000,000,000,000,000,000,000 + 38
If this is evaluated as:
(1,000,000,000,000,000,000,000,000,000 + -1,000,000,000,000,000,000,000,000,000) + 38
You'll get 38. The "38" is so much smaller than the other number that a floating point representation cannot represent the value.
If it is evaluated at:
1,000,000,000,000,000,000,000,000,000 + (-1,000,000,000,000,000,000,000,000,000 + 38)
You'll get 0.
I recommend that you use decimal
s for the calculation.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With