Lua: subtracting decimal numbers doesn't return correct precision

Question

I am using Lua 5.1

print(10.08 - 10.07)

Rather than printing 0.01, above prints 0.0099999999999998.

Any idea how to get 0.01 form this subtraction?

Answer 1

You got 0.01 from the subtraction. It is just in the form of a repeating decimal with a tiny amount of lost precision.

Lua uses the C type double to represent numbers. This is, on nearly every platform you will use, a 64-bit binary floating point value with approximately 23 decimal digits of precision. However, no amount of precision is sufficient to represent 0.01 exactly in binary. The situation is similar to attempting to write 1/3 in decimal.

Furthermore, you are subtracting two values that are very similar in magnitude. That all by itself causes an additional loss of precision.

The solution depends on what your context is. If you are doing accounting, then I would strongly recommend that you not use floating point values to represent account values because these small errors will accumulate and eventually whole cents (or worse) will appear or disappear. It is much better to store accounts in integer cents, and divide by 100 for display.

In general, the answer is to be aware of the issues that are inherent to floating point, and one of them is this sort of small loss of precision. It is easily handled by rounding answers to an appropriate number of digits for display, and never comparing results of calculations for equality.

Some resources for background:

• The semi-official explanation at the Lua Users Wiki
• This great page of IEEE floating point calculators where you can enter values in decimal and see how they are represented, and vice-versa.
• Wiki on IEEE floating point.
• Wiki on floating point numbers in general.
• What Every Computer Scientist Should Know About Floating-Point Arithmetic is the most complete discussion of the fine details.

Edit: Added the WECSSKAFPA document after all. It really is worth the study, although it will likely seem a bit overwhelming on the first pass. In the same vein, Knuth Volume 2 has extensive coverage of arithmetic in general and a large section on floating point. And, since lhf reminded me of its existence, I inserted the Lua wiki explanation of why floating point is ok to use as the sole numeric type as the first item on the list.