Spark: write a CSV with null values as empty columns

Question

I'm using PySpark to write a dataframe to a CSV file like this:

df.write.csv(PATH, nullValue='')

There is a column in that dataframe of type string. Some of the values are null. These null values display like this:

...,"",...

I would like them to be display like this instead:

...,,...

Is this possible with an option in csv.write()?

Thanks!

Answer 1

Easily with emptyValue option setted

emptyValue: sets the string representation of an empty value. If None is set, it use the default value, "".

from pyspark import Row
from pyspark.shell import spark

df = spark.createDataFrame([
    Row(col_1=None, col_2='20151231', col_3='Hello'),
    Row(col_1=2, col_2='20160101', col_3=None),
    Row(col_1=3, col_2=None, col_3='World')
])

df.write.csv(PATH, header=True, emptyValue='')

Output

col_1,col_2,col_3
,20151231,Hello
2,20160101,
3,,World