AnalysisException: CSV data source does not support array<struct<

Question

I am at work and I need immediate help please I have a parquet file and I need to convert it to csv. could u please help me?

error:

AnalysisException: CSV data source does not support array<struct<company:string,dateRange:string,description:string,location:string,title:string>> data type.

I have never worked with this format so I can't even print schema. sorry

printshema:

root
 |-- _id: string (nullable = true)
 |-- Locale: string (nullable = true)
 |-- workExperience: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- company: string (nullable = true)
 |    |    |-- dateRange: string (nullable = true)
 |    |    |-- description: string (nullable = true)
 |    |    |-- location: string (nullable = true)
 |    |    |-- title: string (nullable = true)

Answer 1

You can't save a dataframe which contains column with array/struct type to CSV. You need to cast the column to string before writing.

df.withColumn('workExperience', col('workExperience').cast('string')).write.csv('path')

Answer 2

The parquet schema can be flattened using explode:

df=spark.read.parquet(...)
flattened_df = df.withColumn("tmp", F.explode("workExperience")) \
    .selectExpr("_id", "Locale", "tmp.*")
flattened_df.write.csv(...)