Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

TSQL - make a literal float value

I understand the host of issues in comparing floats, and lament their use in this case - but I'm not the table author and have only a small hurdle to climb...

Someone has decided to use floats as you'd expect GUIDs to be used. I need to retrieve all the records with a specific float value.

sp_help MyTable

-- Column_name  Type    Computed    Length  Prec
-- RandomGrouping   float   no  8   53   

Here's my naive attempt:

--yields no results
SELECT RandomGrouping
FROM MyTable
WHERE RandomGrouping = 0.867153569942739

And here's an approximately working attempt:

--yields 2 records
SELECT RandomGrouping
FROM MyTable
WHERE RandomGrouping BETWEEN 0.867153569942739 - 0.00000001
      AND 0.867153569942739 + 0.00000001

--  0.867153569942739
--  0.867153569942739

In my naive attempt, is that literal a floating point literal? Or is it really a decimal literal that gets converted later?

If my literal is not a floating point literal, what is the syntax for making a floating point literal?

EDIT: Another possibility has occurred to me... it may be that a more precise number than is displayed is stored in this column. It may be impossible to create a literal that represents this number. I will accept answers that demonstrate that this is the case.


EDIT: response to DVK.

TSQL is MSSQLServer's dialect of SQL.

This script works, and so equality can be performed deterministically between float types:

DECLARE @X float
SELECT top 1 @X = RandomGrouping
FROM MyTable
WHERE RandomGrouping BETWEEN 0.839110948199148 - 0.000000000001
  AND 0.839110948199148 + 0.000000000001
  --yields two records
SELECT *
FROM MyTable
WHERE RandomGrouping = @X

I said "approximately" because that method tests for a range. With that method I could get values that are not equal to my intended value.

The linked article doesn't apply because I'm not (intentionally) trying to straddle the world boundaries between decimal and float. I'm trying to work with only floats. This isn't about the non-convertibility of decimals to floats.


Response to Zinglon:

A literal value that can find my records, thanks.

DECLARE @Y binary(8)
SET @Y = 0x3FEAD9FF34076378

SELECT *
FROM MyTable
WHERE convert(binary(8), RandomGrouping) = @Y
like image 416
Amy B Avatar asked Jun 11 '10 16:06

Amy B


People also ask

How do you float a value in SQL?

Syntax. float [ (n) ] Where n is the number of bits that are used to store the mantissa of the float number in scientific notation and, therefore, dictates the precision and storage size. If n is specified, it must be a value between 1 and 53. The default value of n is 53.

Is real and float the same?

Real is a Single Precision Floating Point number, while Float is a Double Precision Floating Point number. The Floating point numbers can store very large or very small numbers than decimal numbers.

What is the difference between float and real in SQL Server?

Real data can hold a value 4 bytes in size, meaning it has 7 digits of precision. Float data can hold 8 bytes, or 15 places after the decimal point. Double data is similar to float, except that it allows for much larger numbers.

What is float format in SQL?

Float is Approximate-number data type, which means that not all values in the data type range can be represented exactly. Decimal/Numeric is Fixed-Precision data type, which means that all the values in the data type range can be represented exactly with precision and scale.


2 Answers

is that literal a floating point literal? Or is it really a decimal literal that gets converted later?

If my literal is not a floating point literal, what is the syntax for making a floating point literal?

The 0.867153569942739 literal in SQL Server is a decimal type, not float. The engine automatically picks appropriate scale and precision to represent the given literal.

To write a literal of the float type you should use the scientific notation, like this: 0.867153569942739E0

This is documented in Constants (Transact-SQL)

decimal constants

decimal constants are represented by a string of numbers that are not enclosed in quotation marks and contain a decimal point. The following are examples of decimal constants:

1894.1204  
2.0  

float and real constants

float and real constants are represented by using scientific notation. The following are examples of float or real values:

101.5E5  
0.5E-2

The sp_describe_first_result_set can tell us the types of columns

EXEC sp_describe_first_result_set N'SELECT 0.867153569942739, 0.867153569942739E0'

It returns numeric(15,15) for the first column and float for the second.


If your column RandomGrouping is indexed, it is much more efficient to use a float literal, because when you wrap RandomGrouping in convert(), an index can't be used.

The following query will use an index:

SELECT *
FROM MyTable
WHERE RandomGrouping = 0.867153569942739E0

The following query will not use an index:

SELECT *
FROM MyTable
WHERE convert(binary(8), RandomGrouping) = @Y
like image 71
Vladimir Baranov Avatar answered Sep 29 '22 10:09

Vladimir Baranov


It is possible that the values are being truncated on display. I'm assuming the column doesn't have a unique constraint on it, otherwise the question would be moot. On my setup, SSMS truncates the more precise value in this script.

create table flt ( f float not null primary key )
insert into flt
select 0.111111111111111
union all
select 0.1111111111111111
select f, cast(f as binary(8)) from flt

Similarly, if these values are distinct you can cast them to binary(8) and identify them based on that value, like this:

select f from flt
where cast(f as binary(8)) = 0x3FBC71C71C71C71C
like image 20
zinglon Avatar answered Sep 29 '22 11:09

zinglon