Interesting float bug in Calculator - why does this occur?

Question

If you enter 1.0000000000000000000000000000001 in the Windows calculator, then repeatedly hit the factorial (n!) button, you get some odd results:

1.0000000000000000000000000000001 [n!]
1                                 [n!]
1                                 [n!]
1                                 [n!]
0.9999999999999999999999999999997 [n!]
0.9999999999999999999999999999998 [n!]
0                                 [n!]
1

Try it yourself - it's rather odd!

My main interest is why we get a zero on the second to last step. What artefact of IEEE floats causes these issues?

Answer 1

The arithmetic in the Windows calculator is horrible. It is not IEEE compliant and, in fact, it ignores all the lessons of the IEEE spec.

Or, at least, it ignores one important lesson: that proper rounding really matters.

Try typing this (in scientific mode):

• 4
• Square root (answer shows up as 2)
• -2 (-8.1648465955514287168521180122928e-39)

This tells us two things about the Windows calculator.

1. It calculates square root to about 40 digits of precision.
2. It fails to round correctly.

40 digits is way more precision than a double, and sometimes that is good, but squareroot(4) shows that without correct rounding that extra 'precision' may be misguided. It is relatively straightforward to correctly round the basic operations (+, -, *, /) and square root. Not doing so is just sloppy.

As to the original question, if you use the "-1" trick to see the true values when the calculator says that the answer is 1 you will see that the number is gradually getting closer to 1. Eventually it's terrible rounding causes it to go less than one. Then something bad happens and it hits zero. I don't know what the something bad is, but given its poor rounding it is hardly surprising that something bad happens.