Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

understanding precision and scale on a property

property name="poiLat" length="60" ormtype="big_decimal" persistent=true precision="16" scale="14" default="0" hint="";

I don't understand precision or scale correctly. Using the property above why would '1' give an error and '2' be accepted? what should I change it to to accept '1'

1 ) -118.27 = error

2) -18.27 = ok

like image 825
Prometheus Avatar asked Jun 13 '12 22:06

Prometheus


People also ask

What does precision and scale mean?

Precision is the number of digits in a number. Scale is the number of digits to the right of the decimal point in a number. For example, the number 123.45 has a precision of 5 and a scale of 2.

What is numeric precision?

Numeric precision refers to the maximum number of digits that are present in the number. ie 1234567.89 has a precision of 9. Numeric scale refers to the maximum number of decimal places. ie 123456.789 has a scale of 3. Thus the maximum allowed value for decimal(5,2) is 999.99.

What is a data type the store's exact numbers with a defined precision and scale?

Decimal Data Type. The decimal data type is an exact numeric data type defined by its precision (total number of digits) and scale (number of digits to the right of the decimal point).

What is column precision?

The precision is the maximum number of digits or characters that are displayed for the data in that column. For nonnumeric data, the precision typically refers to the defined length of the column. The scale refers to the maximum number of digits that are displayed to the right of the decimal point.


1 Answers

Scale refers the number of digits to the right of the decimal place. If you have precision 16 and scale 14, you can only have 2 digits to the left of the decimal place, so

18.12345678901234 = ok 
118.27 = error

Try:

precision="16" scale="13" 

That will allow 118.1234567890123, but that is a lot of decimal places. How many do you really need?

precision="16" scale="4"

Will allow 123456789012.1234

like image 132
mafue Avatar answered Sep 21 '22 11:09

mafue