Menu
I'm on Spark 2.2.0, running on EMR.
I have a big dataframe df (40G or so in compressed snappy files) which is partitioned by keys k1 and k2.
When I query by k1 === v1 or (k1 === v1 && k2 ===v2`), I can see that it's only querying the files in the partition (about 2% of the files).
However if I cache or persist df, suddenly those queries are hitting all the partitions and either blows up memory or is much less performant.
This is a big surprise - is there any way to do caching which preserves the partitoning information
603
cache() is an Apache Spark transformation that can be used on a DataFrame, Dataset, or RDD when you want to perform more than one action. cache() caches the specified DataFrame, Dataset, or RDD in the memory of your cluster's workers.
Advantages for Caching and Persistence of DataFrameTime-efficient – Reusing repeated computations saves lots of time. Execution time – Saves execution time of the job and we can perform more jobs on the same cluster.
The only difference between cache() and persist() is ,using Cache technique we can save intermediate results in memory only when needed while in Persist() we can save the intermediate results in 5 storage levels(MEMORY_ONLY, MEMORY_AND_DISK, MEMORY_ONLY_SER, MEMORY_AND_DISK_SER, DISK_ONLY). Save this answer.
This is to be expected. Spark internal columnar format used for caching is input format agnostic. Once you loaded data there there is no connection to the original input is gone.
The exception here is new data source API [SPARK-22389][SQL] data source v2 partitioning reporting interface, which allows for persisting partitioning information, but it is new in 2.3 and still experimental.
137
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
