Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Incorrect rounding of float when using ToString("F1")

I have a float value: 12345.6489

When I format this using:

(12345.6489f).ToString("F1")

Then I get a result of

12345.7

But this is incorrect, since it should be 12345.6.

Does anyone understand why this might occur? Another hint is that casting to double before I format returns the correct result, and if my float value is a little smaller, for example 1234.6489, then too I get the correct result.

like image 771
L. Desjardins Avatar asked Jan 14 '13 19:01

L. Desjardins


2 Answers

This seems to be related to a question I asked some time ago: Round-twice error in .NET's Double.ToString method

Note that if you call .ToString("G") on your number, it is correctly rounded to 12345.65. If you round the rounded number, to one decimal, the problem occurs.

When I investigated my own question earlier, I also found some examples that couldn't be explained as round-twice errors, so check that thread as well.

Addition: Note that any number that can be represented (exactly) by a float, can also be represented (with a lot of zero bits) by a double. One can use the following trick (which the question also mentions):

float x = 12345.6489f;
string trick = ((double)x).ToString("F1");
like image 173
Jeppe Stig Nielsen Avatar answered Nov 10 '22 09:11

Jeppe Stig Nielsen


Thanks for this question! It is very interesting subject to be engaged in investigations. But I wanted to mention other party of a medal. You asked the following:

(12345.6489f).ToString("F1")

Then I get a result of

12345.7

But this is incorrect, since it should be 12345.6.

Well, I'm wondering how you have figured out what is correct and what is incorrect output of this string formatting routine? Those formatting strings are not supposed to be used to rounding purposes. And documentation clearly says about it:

http://msdn.microsoft.com/en-us/library/dwhawy9k.aspx#FFormatString


Honestly when I first looked at number from your question - the first idea was about rounding algorithm which Hans Passant mentioned in his answer. So, I'm not even surprised that such algorithm was choosen, it is actually pretty intuitive :) I even wouldn't be surprised they would consider plain truncate as algorithm for formatting of floating points numbers. It would be still pretty accurate and valid.

So, despite of the fact that all this is very interesting and looking like a bug/paradox/miracle, this in fact is just our wrong expectations from the function which is designed to do (and actually does pretty well) another thing.

like image 1
SergeyS Avatar answered Nov 10 '22 11:11

SergeyS