Menu
Is there a way to prevent PySpark from creating several small files when writing a DataFrame to JSON file?
If I run:
df.write.format('json').save('myfile.json')
or
df1.write.json('myfile.json')
it creates the folder named myfile and within it I find several small files named part-***, the HDFS way. Is it by any means possible to have it spit out a single file instead?
859
1. Write a Single file using Spark coalesce() & repartition() When you are ready to write a DataFrame, first use Spark repartition() and coalesce() to merge data from all partitions into a single partition and then save it to a file.
In Big Data Management (BDM0, when Spark writes to S3 a complex file of type Parquet, the resulting files may contain an additional _SUCCESS. This file is used by Spark to provide a method to confirm all the partitions have been written correctly. You can control this by setting the Hadoop property: mapreduce.
Each part file will have an extension of the format you write (for example.csv,.json,.txt e.t.c) val df = spark. read. option ("header",true). csv ("address.csv") df. write. csv ("address") This writes multiple part files in address directory.
If not provided, the default limit value is -1. Before we start with an example of Pyspark split function, first let’s create a DataFrame and will use one of the column from this DataFrame to split into multiple columns. Output is shown below for the above code.
PySpark partitionBy () is a function of pyspark.sql.DataFrameWriter class which is used to partition based on column values while writing DataFrame to Disk/File system. When you write PySpark DataFrame to disk by calling partitionBy (), PySpark splits the records based on the partition column and stores each partition data into a sub-directory.
PySpark Split Column into multiple columns. Following is the syntax of split() function. In order to use this first you need to import pyspark.sql.functions.split. Syntax: pyspark.sql.functions.split(str, pattern, limit=-1) Parameters: str – a string expression to split; pattern – a string representing a regular expression.
Well, the answer to your exact question is coalesce function. But as already mentioned it is not efficient at all as it will force one worker to fetch all data and write it sequentially.
df.coalesce(1).write.format('json').save('myfile.json')
P.S. Btw, the result file is not a valid json file. It is a file with a json object per line.
103
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
