Spark: How to save a dataframe with headers?

Question

dataframe.saveasTextFile, saves only the data in a delimited format. How do I save the dataframe with headers in JAVA.

sourceRufFrame.toJavaRDD().map(new TildaDelimiter()).coalesce(1, true).saveAsTextFile(targetSrcFilePath);

Answer 1

If you want to save as csv file, i would suggest using spark-csv package. You can save your dataframe simply with spark-csv as below with header.

dataFrame.write
  .format("com.databricks.spark.csv")
  .option("header", "true")
  .option("delimiter",<your delimiter>)
  .save(output)

You can refer below link, for further information: https://github.com/databricks/spark-csv

Spark-csv has maven dependency.

Answer 2

With Spark 2.x,

df.write.option("header", "true").csv("path")

Cheers