Decimal Type with Precision Equivalent in Spark SQL

Question

What is Spark SQL datatype Equivalent to DecimalType(2,9) in SQL?

For example: print(column.dataType==X) => should give me True. Where Column's datatype in SQL is DecimalType(2,9)

Tried: X= DataTypes.createDecimalType(2,9), works fine

I am looking for a generalized DecimalType Class to filter all DecimalType columns from the dataframe irrespective of the precision and scale.

Answer 1

Each DecimalType type is an instance of DecimalType class:

from pyspark.sql.types import DecimalType

df = (spark
  .createDataFrame(["1.32"], "string")
  .selectExpr("CAST(value AS DECIMAL(4, 2)) AS value"))

isinstance(df.schema["value"].dataType, DecimalType)
# True

import org.apache.spark.sql.types._

val df = Seq("1.32").toDF("value")  
  .selectExpr("CAST(value AS DECIMAL(4, 2)) AS value")

df.schema("value").dataType match {
  case _: DecimalType => true
  case _ => false
}
// Boolean = true

Of course, you'll never encounter DECIMAL(2,9), as it is not possible to have scale greater than precision.