Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Rounding quirk in JavaScript or IEEE-754?

I've come across a curious issue in one of my unit tests where I'm getting unexpected rounding results in JavaScript:

(2.005).toFixed(2)
// produces "2.00"

(2.00501).toFixed(2)
// produces "2.01"

Initially I suspected this was a Webkit only issue but it repros in Gecko which implies to me that it is an expected side effect of either ECMA-262 or IEEE-754. I'm assuming the binary representation of 2.005 is ever so slightly less? Or does ECMA-262 specify a round-to-even methodology for toFixed?

Anyone care to shed some insight as to what is happening under the hood just to give me peace of mind?

Update: thanks for the comments.

I should add, one of the things that made me a little nervous was the comments found in a quick search in Webkit dtoa.cpp which seemed to imply that there were multiple paths to rounding and the devs weren't really sure how it worked, including a related FIXME:

https://trac.webkit.org/browser/trunk/Source/WTF/wtf/dtoa.cpp#L1110

Also, not that it means much but IE9 rounds it as I expected, implying that it either isn't part of ECMA-262 or they have a bug.

like image 657
mckamey Avatar asked Feb 19 '23 14:02

mckamey


1 Answers

If the specification hasn't changed since Rev. 6 of the ECMA 262 draft (edition 5.1, March 2011), (2.005).toFixed(2) must return the string "2.00", since a "Number value" is a

primitive value corresponding to a double-precision 64-bit binary format IEEE 754 value

and the interpretation of numeric literals is specified in 7.8.3 and 8.5 to conform to IEEE 754 "round to nearest" mode (with ties rounded to even significand), which for 2.005 results in the value

x = 4514858626438922 * 2^(-51) = 2.00499999999999989341858963598497211933135986328125

In section 15.7.4.5 which deals with toFixed, the relevant step 8. a. is:

Let n be an integer for which the exact mathematical value of n ÷ 10fx is as close to zero as possible. If there are two such n, pick the larger n.

and 2.00 - x is closer to zero than 2.01 - x, so n must be 200 here. The conversion to a string proceeds then in the natural way.

Also, not that it means much but IE9 rounds it as I expected, implying that it either isn't part of ECMA-262 or they have a bug.

A bug. Maybe they tried to go the easy way and multiply with 10^digits and round. x*100 is exactly 200.5, so that would produce a string of "2.01".

like image 130
Daniel Fischer Avatar answered Mar 03 '23 04:03

Daniel Fischer