Convert value depending on a type in SparkSQL via case matching of type

Question

Is it possible to match a parametric type in Scala? Lets say I have a function that receives two parameters: a value and a type. I would like to use pattern matching to do a type conversion.

Something like this:

datatype match {
  case IntegerType => return value.toInt
  case FloatType => return value.toFloat
  case StringType => return value
  case DecimalType(_,_) => return BigDecimal(value) // this is not working
  case _ => return strrepr
}

Here DecimalType accepts two parameters to specify precision the required precision. It can be for example:

org.apache.spark.sql.types.DecimalType = DecimalType(10,2)

I have tried several options and nothing seems to be working:

• For case DecimalType => return BigDecimal(value) I get:

  error: pattern type is incompatible with expected type;
     found   : org.apache.spark.sql.types.DecimalType.type
     required: org.apache.spark.sql.types.DataType
     Note: if you intended to match against the class, try `case DecimalType(_,_)`
  

• For case DecimalType(_,_) => return BigDecimal(value) I get:

  error: result type Boolean of unapply defined in method unapply in object DecimalType does not conform to Option[_] or Boolean
  

• For case DecimalType[_,_] => return BigDecimal(value) I get:

  error: org.apache.spark.sql.types.DecimalType does not take type parameters
  

Answer 1

Turns out that DecimalType only pattern matches with zero arguments:

  case DecimalType() => ...

If you need the precision and scale, you must define the type of the case and manually extract them:

datatype match {
  case dt: DecimalType =>
    val precision = dt.precision
    val scale = dt.scale
    ...

Answer 2

The problem is the use of the return in your code. You said you use this code snippet in a function somewhere. What is the return type of that function? Obviously, you intend that sometimes it is Integer, sometimes String, sometimes BigDecimal; but if you use return, it will look to the type of the returned object to determine the return type of the function. In general, you should strongly avoid using return in Scala code. The last evaluated value in the function body is returned. The only case for using a return is when you want to force returning a value somewhere else in the function body. But still, a better way would be to save the return object in a variable and just evaluate that variable in the last line of your function body. And never use return!

Without return it works

scala> val datatype = DecimalType(10, 2)
datatype: org.apache.spark.sql.types.DecimalType = DecimalType(10,2)

scala> val value = BigDecimal(10)
value: scala.math.BigDecimal = 10

scala> datatype match {case DecimalType(_,_) => value}
res150: scala.math.BigDecimal = 10

** Problems with return **

scala> def test = {datatype match {case DecimalType(_,_) => return value}}
<console>:138: error: method test has return statement; needs result type
       def test = {datatype match {case DecimalType(_,_) => return value}}

scala> def test:BigDecimal = {datatype match {case DecimalType(_,_) => return value}}
test: BigDecimal

scala> def test:DataType = {datatype match {case DecimalType(_,_) => return value}}
<console>:138: error: type mismatch;
 found   : scala.math.BigDecimal
 required: org.apache.spark.sql.types.DataType
       def test:DataType = {datatype match {case DecimalType(_,_) => return value}}

scala> def test3 = {datatype match {case DecimalType(_,_) => value}}
test3: scala.math.BigDecimal