Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Difference between Spark RDD's take(1) and first()

I used to think that rdd.take(1) and rdd.first() are exactly the same. However I began to wonder if this is really true after my colleague pointed me to Spark's officiation documentation on RDD:

first(): Return the first element in this RDD.

take(num): Take the first num elements of the RDD. It works by first scanning one partition, and use the results from that partition to estimate the number of additional partitions needed to satisfy the limit.

My questions are:

  1. Is the underlying implementation of first() the same as take(1)?
  2. Suppose rdd1 and rdd2 are constructed from the same csv, can I safely assume that rdd1.take(1) and rdd2.first() will always return the same result, i.e., the first row of the csv? What if rdd1 and rdd2 are partitioned differently?
like image 630
Ida Avatar asked May 28 '16 04:05

Ida


People also ask

What is Take () in Spark?

take (num: int) → List[T][source] Take the first num elements of the RDD. It works by first scanning one partition, and use the results from that partition to estimate the number of additional partitions needed to satisfy the limit. Translated from the Scala implementation in RDD#take().

What does First () do in Spark?

In Spark, the First function always returns the first element of the dataset. It is similar to take(1).

What is difference between take and collect in Spark?

collect() : It will show the content and metadata of the dataframe. df. take() : shows content and structure/metadata for a limited number of rows for a very large dataset.


1 Answers

Infact first is implemented in terms of take.

Following is taken from spark's source of RDD.scala. first calls take(1) and returns the first element if found.

  def first(): T = withScope {
    take(1) match {
      case Array(t) => t
      case _ => throw new UnsupportedOperationException("empty collection")
    }
  }

take(num) tries to take num elements from starting from RDD's 0th partition (if you consider 0 based indexes). So the behavior of take(1) and first will be identical.

Even the spark programming guide confirms this.

About your second question: it depends what you mean when you say partitioned differently. If you are calling sc.textFile("/path/to/file") with or without numPartitions, it wouldn't matter because 0th partition will always be 0th partition. So Yes, you can assume that they will have the same first element.

EDIT: Partitions in RDD are ordered, the physical first line in your CSV will end up in the 0th partition on RDD. And take(1) and first both will return that first row of 0th partition.

like image 75
Pranav Shukla Avatar answered Sep 28 '22 05:09

Pranav Shukla