How can I find the size of a RDD

Question

I have RDD[Row], which needs to be persisted to a third party repository. But this third party repository accepts of maximum of 5 MB in a single call.

So I want to create partition based on the size of the data present in RDD and not based on the number of rows present in RDD.

How can I find the size of a RDD and create partitions based on it?

Answer 1

As Justin and Wang mentioned it is not straight forward to get the size of RDD. We can just do a estimate.

We can sample a RDD and then use SizeEstimator to get the size of sample. As Wang and Justin mentioned, based on the size data sampled offline, say, X rows used Y GB offline, Z rows at runtime may take Z*Y/X GB

Here is the sample scala code to get the size/estimate of a RDD.

I am new to scala and spark. Below sample may be written in a better way

def getTotalSize(rdd: RDD[Row]): Long = {   // This can be a parameter   val NO_OF_SAMPLE_ROWS = 10l;   val totalRows = rdd.count();   var totalSize = 0l   if (totalRows > NO_OF_SAMPLE_ROWS) {     val sampleRDD = rdd.sample(true, NO_OF_SAMPLE_ROWS)     val sampleRDDSize = getRDDSize(sampleRDD)     totalSize = sampleRDDSize.*(totalRows)./(NO_OF_SAMPLE_ROWS)   } else {     // As the RDD is smaller than sample rows count, we can just calculate the total RDD size     totalSize = getRDDSize(rdd)   }    totalSize }  def getRDDSize(rdd: RDD[Row]) : Long = {     var rddSize = 0l     val rows = rdd.collect()     for (i <- 0 until rows.length) {        rddSize += SizeEstimator.estimate(rows.apply(i).toSeq.map { value => value.asInstanceOf[AnyRef] })     }      rddSize }