Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What is the nature of the round off error here?

Can someone help me unpack what exactly is going on under the hood here?

>>> 1e16 + 1.
1e+16
>>> 1e16 + 1.1
1.0000000000000002e+16

I'm on 64-bit Python 2.7. For the first, I would assume that since there is only a precision of 15 for float that it's just round-off error. The true floating-point answer might be something like

10000000000000000.999999....

And the decimal just gets lopped of. But the second result makes me question this understanding and can't 1 be represented exactly? Any thoughts?

[Edit: Just to clarify. I'm not in any way suggesting that the answers are "wrong." Clearly, they're right, because, well they are. I'm just trying to understand why.]

like image 720
jseabold Avatar asked Feb 25 '13 03:02

jseabold


1 Answers

It's just rounding as close as it can.

1e16 in floating hex is 0x4341c37937e08000.

1e16+2 is 0x4341c37937e08001.

At this level of magnitude, the smallest difference in precision that you can represent is 2. Adding 1.0 exactly rounds down (because typically IEEE floating point math will round to an even number). Adding values larger than 1.0 will round up to the next representable value.

like image 137
StilesCrisis Avatar answered Oct 18 '22 23:10

StilesCrisis