Menu
I get confusing messages when searching and reading answers on the internet on this subject. Anyone can share their experience? I know for a fact that gzipped csv is not, but maybe file internal structures for Parquet are such that it is totally different case for Parquet vs csv?
323
Yes, Parquet files are splittable. S3 supports positioned reads (range requests), which can be used to read only selected portions of the input file (object).
Gzip is supported by Spark and by Parquet, but not like this. Apache Parquet is a free and open-source column-oriented data storage format of the Apache Hadoop ecosystem.
Each block in the parquet file is stored in the form of row groups. So, data in a parquet file is partitioned into multiple row groups. These row groups in turn consists of one or more column chunks which corresponds to a column in the dataset. The data for each column chunk is then written in the form of pages.
Parquet has higher execution speed compared to other standard file formats like Avro,JSON etc and it also consumes less disk space in compare to AVRO and JSON.
Parquet files with GZIP compression are actually splittable. This is because of the internal layout of Parquet files. These are always splittable, independent of the used compression algorithm.
This fact is mainly due to the design of Parquet files that divided in the following parts:
You can find a more detailed explanation here: https://github.com/apache/parquet-format#file-format
97
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
