Apache Spark: Get number of records per partition

Question

I want to check how can we get information about each partition such as total no. of records in each partition on driver side when Spark job is submitted with deploy mode as a yarn cluster in order to log or print on the console.

Answer 1

I'd use built-in function. It should be as efficient as it gets:

import org.apache.spark.sql.functions.spark_partition_id  df.groupBy(spark_partition_id).count 

Answer 2

You can get the number of records per partition like this :

df   .rdd   .mapPartitionsWithIndex{case (i,rows) => Iterator((i,rows.size))}   .toDF("partition_number","number_of_records")   .show 

But this will also launch a Spark Job by itself (because the file must be read by spark to get the number of records).

Spark could may also read hive table statistics, but I don't know how to display those metadata..