Spark: what options can be passed with DataFrame.saveAsTable or DataFrameWriter.options?

Question

Neither the developer nor the API documentation includes any reference about what options can be passed in DataFrame.saveAsTable or DataFrameWriter.options and they would affect the saving of a Hive table.

My hope is that in the answers to this question we can aggregate information that would be helpful to Spark developers who want more control over how Spark saves tables and, perhaps, provide a foundation for improving Spark's documentation.

Answer 1

The reason you don't see options documented anywhere is that they are format-specific and developers can keep creating custom write formats with a new set of options.

However, for few supported formats I have listed the options as mentioned in the spark code itself:

• CSVOptions
• JDBCOptions
• JSONOptions
• ParquetOptions
• TextOptions
• OrcOptions
• AvroOptions

Answer 2

Take a look at the options file in GitHub the class "DeltaOptions'

Currently, supported options include:

• replaceWhere
• mergeSchema
• overwriteSchema
• userMetadata
• partitionOverwriteMode (DYNAMIC, STATIC)
• maxFilesPerTrigger
• excludeRegex
• ignoreFileDeletion
• ignoreChanges
• ignoreDeletes
• skipChangeCommits
• failOnDataLoss
• optimizeWrite
• dataChange
• queryName
• checkpointLocation
• path
• startingVersion
• startingTimestamp
• timestampAsOf
• versionAsOf