Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Using CType to single with VB.net returns extra digit

Tags:

vb.net

I am using VB.NET 2010. I ran into an issue that at first I thought was a database issue since the value was being pulled from a DB2 table. However, it appears to be in .NET itself. I determined this by running the following two lines of code:

    Dim sAmount As Single
    sAmount = CType("212639.04", Single)

When I then look at sAmount, the value is 212639.047.

My question: Where is the 7 coming from?

like image 547
David Hodgkins Avatar asked Nov 02 '22 07:11

David Hodgkins


1 Answers

The trouble is you're using Single which is a floating-pont data type, which doesn't work well with fractions that aren't binary fractions.

Floating-Point Expressions Do Not Compare as Equal

When you work with floating-point numbers (Single Data Type (Visual Basic) and Double Data Type (Visual Basic)), remember that they are stored as binary fractions. This means they cannot hold an exact representation of any quantity that is not a binary fraction (of the form k / (2 ^ n) where k and n are integers). For example, 0.5 (= 1/2) and 0.3125 (= 5/16) can be held as precise values, whereas 0.2 (= 1/5) and 0.3 (= 3/10) can be only approximations.

Because of this imprecision, you cannot rely on exact results when you operate on floating-point values. In particular, two values that are theoretically equal might have slightly different representations.

To compare floating-point quantities

Calculate the absolute value of their difference by using the Abs method of the 
Math class in the System namespace.

Determine an acceptable maximum difference, such that you can consider the two 
quantities to be equal for practical purposes if their difference is no larger.

Compare the absolute value of the difference to the acceptable difference.

Alternatively double will return a number with greater precision.

like image 67
tinstaafl Avatar answered Nov 15 '22 07:11

tinstaafl