In Scala / Spark, how to convert empty string, like " ", to "NULL" ? need to trim it first and then convert to "NULL". Thanks.
dataframe.na.replace("cut", Map(" " -> "NULL")).show //wrong
You can create a simple function to do it. First a couple of imports:
import org.apache.spark.sql.functions.{trim, length, when}
import org.apache.spark.sql.Column
and the definition:
def emptyToNull(c: Column) = when(length(trim(c)) > 0, c)
Finally a quick test:
val df = Seq(" ", "foo", "", "bar").toDF
df.withColumn("value", emptyToNull($"value"))
which should yield following result:
+-----+
|value|
+-----+
| null|
| foo|
| null|
| bar|
+-----+
If you want to replace empty string with string "NULL
you can add otherwise
clause:
def emptyToNullString(c: Column) = when(length(trim(c)) > 0, c).otherwise("NULL")
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With