how to write spark custom data source based on FileFormat

Question

i saw the spark avro datasource is implemented based on FileFormat interface. Is there any documentation about how to write spark custom datasource based on FileFormat? Up to now i can't find any(except the source code from spark avro).

Thank you!

Answer 1

Here is an example of a simple file-based spark datasource: https://hackernoon.com/extending-our-spark-sql-query-engine-5f4a088de986

Here's a couple examples that implement the Data Sources API, as well: * https://github.com/databricks/spark-csv * https://github.com/databricks/spark-avro