Use Spark when function without otherwise but keep column values

Question

I frequently find myself replacing values in columns using

when($"myCol".isNull,myCrazyFunction).otherwise($"myCol")

To me the .otherwise($"myCol") feels kind of redundant.

Is there a better way to replace some values under some condition and otherwise just leave everything as it is without using the otherwise?

Answer 1

I think you can use coalesce() for that.

select(coalesce($"myCol", myCrazyFunction))

Just remember that myCrazyFunction should return a Column type.