I'm using spark 2.4.1 and scala, and trying to write DF to csv file. it seems that in case of null values ,the csv contains "". Is it possible to remove those empty quotes?
val data = Seq(
Row(1, "a"),
Row(5, "z"),
Row(5, null)
)
val schema = StructType(
List(
StructField("num", IntegerType, true),
StructField("letter", StringType, true)
)
)
var df = spark.createDataFrame(
spark.sparkContext.parallelize(data),
schema
)
df.write.csv("location/")
The output seems like:
1,a
5,z
5,""
And I want it will be:
1,a
5,z
5,
What should I do?
Thanks!
You can use options of the writer see CSV specific options(SaveMode is not related to answer);
df.write
.option("nullValue", null)
.mode(SaveMode.Overwrite)
.csv("location/")
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With