How to save file in Feather format\storage from Spark?

Question

Is it possible to export data-frame from Apache Spark to feather (https://github.com/wesm/feather) file?

Answer 1

Not sure, you can do it directly, but you can transform first the Spark Dataframe (on pyspark) to a pandas and store it the to Feather:

pandas_df = spark_df.toPandas()

feather.write_feather(pandas_df, 'example_feather')

But I afraid, this will have an impact on the performance.