PySpark truncate a decimal

Question

I'm working in pySpark and I have a variable LATITUDE that has a lot of decimal places. I need to create two new variables from this, one that is rounded and one that is truncated. Both to three decimal places.

What is the simplest way to truncate a value?

For rounding, I did:

raw_data = raw_data.withColumn("LATITUDE_ROUND", round(raw_data.LATITUDE, 3))

This seems to work, but let me know if there is a better way.

Answer 1

Try:

>>> from pyspark.sql.functions import pow, lit
>>> from pyspark.sql.types import LongType
>>>
>>> num_places = 3
>>> m = pow(lit(10), num_places).cast(LongType())
>>> df = sc.parallelize([(0.6643, ), (0.6446, )]).toDF(["x"])
>>> df.withColumn("trunc", (col("x") * m).cast(LongType()) / m).

Answer 2

You could use the floor() function. So (without testing) I'd suggest:

raw_data = raw_data.withColumn("LATITUDE_TRUNCATED", floor(raw_data.LATITUDE))

But watch out for negative values - as in https://math.stackexchange.com/questions/344815/how-do-the-floor-and-ceiling-functions-work-on-negative-numbers