Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why does MS-SQL-Server (all versions) cast 1.0/12.0 as numeric(8,6)?

I'm trying to explain a 101.10 difference in value.

303'300'000/12 is about 25'275'000.
However, according to MS-SQL, it's 25'274'898.90.

Consider this (dumbed down) SQL-statement:

SELECT 
     303300000.00/12.0 AS a 
    ,1.0/12.0*303300000.00 AS b
    ,1.0/12.0 AS omg 
    ,1.0/CAST(12.0 AS float) AS expected 
    ,0.083333*303300000.0 AS r1 
    ,0.083333333333333300 * 303300000.0 AS r2

astounding results

I thought since I wrote 1.0/12.0 it would cast as float
(which is itselfs dumb, but that's another story),
but apparently, it's decimal (8,6)

CREATE VIEW dbo._Test
AS
SELECT 1.0/12.0 as xxx



SELECT 
     COLUMN_NAME
    ,DATA_TYPE
    ,NUMERIC_PRECISION
    ,NUMERIC_SCALE
FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = '_Test' 


DROP VIEW dbo._Test

Is there any reason to this madness ?
How does it determine the precision ?
Is there any notation to force float, without the need for a cast statement ?

If I run the same query on PostgreSQL, pg does it right...

postgre does it right

like image 410
Stefan Steiger Avatar asked Jan 28 '23 15:01

Stefan Steiger


1 Answers

This is explained in the documentation: Precision, scale, and Length (Transact-SQL)

Specifically it states:

The operand expressions are denoted as expression e1, with precision p1 and scale s1, and expression e2, with precision p2 and scale s2. The precision and scale for any expression that is not decimal is the precision and scale defined for the data type of the expression.

Operation     Result precision                        Result scale *
e1 + e2       max(s1, s2) + max(p1-s1, p2-s2) + 1     max(s1, s2)
e1 - e2       max(s1, s2) + max(p1-s1, p2-s2) + 1     max(s1, s2)
e1 * e2       p1 + p2 + 1                             s1 + s2
e1 / e2       p1 - s1 + s2 + max(6, s1 + p2 + 1)      max(6, s1 + p2 + 1)

The important part here is the last one. In your case, you have a decimal(2,1) and a decimal(3,1). For precision, this results in:

 2 - 1 + 1 + max(6,1 + 3 + 1) = 2 + max(6,5) = 2 + 6 = 8

For the scale, we get:

max(6,1+3+1) = max(6,5) = 6

Take the resulting values and as a result you get a decimal(8,6).

like image 120
Larnu Avatar answered Jan 30 '23 03:01

Larnu