Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Result of casting double to int is wrong

There seems to be some kind of obscure rounding error when I run the following code:

int roundedTotal = (int)(PriorityJob * 100.0);

Initially PriorityJob = 1.4 and roundedTotal is undefined. Evaluating PriorityJob * 100.0 at that point gives 140. Afterwards roundedTotal = 139.

Apparently, 140.0 is being interpreted as 139.99999. Is this a deficiency in the floating point engine? I have never seen anything like it.

like image 846
Michael Sandler Avatar asked Sep 14 '12 12:09

Michael Sandler


1 Answers

Just about every modern computer uses a binary representation for floating-point numbers.

Just as 1/3 = 0.33333333... can't be represented exactly as a decimal fraction, so 1/10 (and hence most non-integer decimal values, including 1.4) can't be represented exactly as a binary fraction. It will instead be represented by the nearest representable value, which may be slightly more or less than the "true" value.

You might want to round to the nearest integer instead: (int)(PriorityJob * 100.0 + 0.5)

like image 51
Mike Seymour Avatar answered Oct 16 '22 16:10

Mike Seymour