Spark UDF error - Schema for type Any is not supported

Question

I’m trying to create a udf that will replace negative values in a column with 0.

My dataframe is – called df, and contains one column called avg_x. This is my code for creating a udf

val noNegative = udf {(avg_acc_x: Double) => if(avg_acc_x < 0) 0 else "avg_acc_x"}

I get this error

java.lang.UnsupportedOperationException: Schema for type Any is not supported

df.printSchema returns

|-- avg_acc_x: double (nullable = false) 

so I don’t understand why this error is occurring?

Answer 1

This error basically occurs when the Analyzer cannot resolve the type of udf properly. That is, the types are mixed (Int and String).

Therefore,

if(avg_acc_x < 0) 0 else avg_acc_x (Integer Type)

or

if(avg_acc_x < 0) "0" else "avg_acc_x" (String Type)

should resolve this exception.

Answer 2

It's because of the else returning a String: "avg_acc_x". Take away the quotes:

val noNegative = udf {(avg_acc_x: Double) => if(avg_acc_x < 0) 0 else avg_acc_x}