Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I fix my output for floating-point imprecision?

I am doing some float manipulation and end up with the following numbers:

-0.5
-0.4
-0.3000000000000000004
-0.2000000000000000004
-0.1000000000000000003
1.10E-16
0.1
0.2
0.30000000000000000004
0.4
0.5

The algorithm is the following:

var inc:Number = nextMultiple(min, stepSize);
trace(String(inc));

private function nextMultiple(x:Number, y:Number) {
    return Math.ceil(x/y)*y;
}

I understand the fact the float cannot always be represented accurately in a byte. e.g 1/3. I also know my stepsize being 0.1. If I have the stepsize how could I get a proper output?

The strange thing is that its the first time I've encountered this type of problem. Maybe I dont play with float enough.

like image 635
coulix Avatar asked Dec 14 '22 05:12

coulix


2 Answers

A language agnostic solution would be to store your numbers as an integer number of steps, given that you know your step size, instead of as floats.

A non-language agnostic solution would be to find out what your language's implementation of printf is.

printf ("float: %.1f\n", number);
like image 149
Bill the Lizard Avatar answered Dec 26 '22 17:12

Bill the Lizard


The limited floating point precision of binary numbers is your problem, as you recognize. One way around this is not to do floating point math. Translate the problem to integers, then translate back for the output.

like image 25
brian d foy Avatar answered Dec 26 '22 18:12

brian d foy