Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why is my number being rounded incorrectly?

This feels like the kind of code that only fails in-situ, but I will attempt to adapt it into a code snippet that represents what I'm seeing.

float f = myFloat * myConstInt; /* Where myFloat==13.45, and myConstInt==20 */
int i = (int)f;
int i2 = (int)(myFloat * myConstInt);

After stepping through the code, i==269, and i2==268. What's going on here to account for the difference?

like image 795
izb Avatar asked Mar 24 '10 16:03

izb


2 Answers

Float math can be performed at higher precision than advertised. But as soon as you store it in float f, that extra precision is lost. You're not losing that precision in the second method until, of course, you cast the result down to int.

Edit: See this question Why differs floating-point precision in C# when separated by parantheses and when separated by statements? for a better explanation than I probably provided.

like image 80
Anthony Pegram Avatar answered Sep 27 '22 16:09

Anthony Pegram


Because floating point variables are not infinitely accurate. Use a decimal if you need that kind of accuracy.

Different rounding modes may also play into this issue, but the accuracy problem is the one you're running into here, AFAIK.

like image 25
technophile Avatar answered Sep 27 '22 16:09

technophile