Spark-Csv Write quotemode not working

Question

I am trying to write a DataFrame as a CSV file using Spark-CSV (https://github.com/databricks/spark-csv)

I am using the command below

res1.write.option("quoteMode", "NONE").format("com.databricks.spark.csv").save("File")

But my CSV file is always written as

"London"
"Copenhagen"
"Moscow"

instead of

London
Copenhagen
Moscow

Answer 1

this problem bothers me for a long time until I read this: Adding custom Delimiter adds double quotes in the final spark data frame CSV outpu

This is a standard CSV feature. If there's an occurrence of delimiter in the actual data (referred to as Delimiter Collision), the field is enclosed in quotes. You can try df.write.option("delimiter" , somechar) where somechar should be a character that doesn't occur in your data.

You can just concat multiple columns into one and use a delimiter that is not in your data

Answer 2

Yes. The way to turn off the default escaping of the double quote character (") with the backslash character (\), you must add an .option() method call with just the right parameters after the .write() method call. The goal of the option() method call is to change how the csv() method "finds" instances of the "quote" character. To do this, you must change the default of what a "quote" actually means; i.e. change the character sought from being a double quote character (") to a Unicode "\u0000" character (essentially providing the Unicode NUL character which won't ever occur within a well formed JSON document).

val dataFrame =
  spark.sql("SELECT * FROM some_table_with_a_json_column")
val unitEmitCsv =
  dataframe
    .write
    .option("header", true)
    .option("delimiter", "\t")
    .option("quote", "\u0000") //magic is happening here
    .csv("/FileStore/temp.tsv")

This was only one of several lessons I learned attempting to work with Apache Spark and emitting .csv files. For more information and context on this, please see the blog post I wrote titled "Example Apache Spark ETL Pipeline Integrating a SaaS".

Answer 3

The double quoting of the text can be removed by setting the quoteAll option to false

dataframe.write
 .option("quoteAll", "false")
 .format("csv")

This example is as per Spark 2.1.0 with out using the databricks lib.