Python: Float to Decimal conversion and subsequent shifting may give wrong result

Question

When I convert a float to decimal.Decimal in Python and afterwards call Decimal.shift it may give me completely wrong and unexpected results depending on the float. Why is this the case?

Converting 123.5 and shifting it:

from decimal import Decimal

a = Decimal(123.5)
print(a.shift(1))  # Gives expected result

The code above prints the expected result of 1235.0.

If I instead convert and shift 123.4:

from decimal import Decimal

a = Decimal(123.4)
print(a.shift(1))  # Gives UNexpected result

it gives me 3.418860808014869689941406250E-18 (approx. 0) which is completely unexpected and wrong.

Why is this the case?

---

Note: I understand the floating-point imprecision because of the representation of floats in memory. However, I can't explain why this should give me such a completely wrong result.

Edit: Yes, in general it would be best to not convert the floats to decimals but convert strings to decimals instead. However, this is not the point of my question. I want to understand why the shifting after float conversion gives such a completely wrong result. So if I print(Decimal(123.4)) it gives 123.40000000000000568434188608080148696899414062 so after shifting I would expect it to be 1234.0000000000000568434188608080148696899414062 and not nearly zero.

Answer 1

You need to change the Decimal constructor input to use strings instead of floats.

a = Decimal('123.5')
print(a.shift(1))

a = Decimal('123.4')
print(a.shift(1))

or

a = Decimal(str(123.5))
print(a.shift(1))

a = Decimal(str(123.4))
print(a.shift(1))

The output will be as expected.

>>> 1235.0
>>> 1234.0

Decimal instances can be constructed from integers, strings, floats, or tuples. Construction from an integer or a float performs an exact conversion of the value of that integer or float.

For floats, Decimal calls Decimal.from_float()

Note that Decimal.from_float(0.1) is not the same as Decimal('0.1'). Since 0.1 is not exactly representable in binary floating point, the value is stored as the nearest representable value which is 0x1.999999999999ap-4. The exact equivalent of the value in decimal is 0.1000000000000000055511151231257827021181583404541015625.

Internally, the Python decimal library converts a float into two integers representing the numerator and denominator of a fraction that yields the float.

n, d = abs(123.4).as_integer_ratio()

It then calculates the bit length of the denominator, which is the number of bits required to represent the number in binary.

k = d.bit_length() - 1

And then from there the bit length k is used to record the coefficient of the decimal number by multiplying the numerator * 5 to the power of the bit length of the denominator.

coeff = str(n*5**k)

The resulting values are used to create a new Decimal object with constructor arguments of sign, coefficient, and exponent using this values.

For the float 123.5 these values are

>>> 1 1235 -1

and for the float 123.4 these values are

1 123400000000000005684341886080801486968994140625 -45

So far, nothing is amiss.

However when you call shift, the Decimal library has to calculate how much to pad the number with zeroes based on the shift you've specified. To do this internally it takes the precision subtracted by length of the coefficient.

amount_to_pad = context.prec - len(coeff)

The default precision is only 28 and with a float like 123.4 the coefficient becomes much longer than the default precision as noted above. This creates a negative amount to pad with zeroes and makes the number very tiny as you noted.

A way around this is to increase the precision to the length of the exponent + the length of the number you started with (45 + 4).

from decimal import Decimal, getcontext

getcontext().prec = 49

a = Decimal(123.4)
print(a)
print(a.shift(1))

>>> 123.400000000000005684341886080801486968994140625
>>> 1234.000000000000056843418860808014869689941406250

The documentation for shift hints that the precision is important for this calculation:

The second operand must be an integer in the range -precision through precision.

However it does not explain this caveat for floats that don't play nice with memory limitations.

I would expect this to raise some kind of error and prompt you to change your input or increase the precision, but at least you know!

@MarkDickinson noted in a comment above that you can view this Python bug tracker for more information: https://bugs.python.org/issue7233