Prevent DataFrame.partitionBy() from removing partitioned columns from schema

Question

I am partitioning a DataFrame as follows:

df.write.partitionBy("type", "category").parquet(config.outpath) 

The code gives the expected results (i.e. data partitioned by type & category). However, the "type" and "category" columns are removed from the data / schema. Is there a way to prevent this behaviour?

Answer 1

I can think of one workaround, which is rather lame, but works.

import spark.implicits._  val duplicated = df.withColumn("_type", $"type").withColumn("_category", $"category") duplicated.write.partitionBy("_type", "_category").parquet(config.outpath) 

I'm answering this question in hopes that someone would have a better answer or explanation than what I have (if OP has found a better solution), though, since I have the same question.

Answer 2

In general, Ivan's answer is a fine cludge. BUT...

If you are strictly reading and writing in spark, you can just use the basePath option when reading your data.

https://spark.apache.org/docs/2.2.0/sql-programming-guide.html#partition-discovery

By passing path/to/table to either SparkSession.read.parquet or SparkSession.read.load, Spark SQL will automatically extract the partitioning information from the paths.

Example:

     val dataset = spark       .read       .format("parquet")       .option("basePath", hdfsInputBasePath)       .load(hdfsInputPath)