What does withReplacement do, if specified for sample against a Spark Dataframe

Question

Reading the spark documentation: http://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.DataFrame.sample

There is this boolean parameter withReplacement without much explanation.

sample(withReplacement, fraction, seed=None)

What is it and how do we use it?

Answer 1

The parameter withReplacement controls the Uniqueness of sample result. If we treat a Dataset as a bucket of balls, withReplacement=true means, taking a random ball out of the bucket and place it back into it. that means, the same ball can be picked up again.

Assuming all unique elements in a Dataset:

• withReplacement=true, same element can be produced more than once as the result of sample.

• withReplacement=false, each element of the dataset will be sampled only once.

     import spark.implicits._
  
      val df = Seq(1, 2, 3, 5, 6, 7, 8, 9, 10).toDF("ids")
  
      df.show()
  
      df.sample(true, 0.5, 5)
        .show
  
      df.sample(false, 0.5, 5)
        .show
  

  Result

  +---+
  |ids|
  +---+
  |  1|
  |  2|
  |  3|
  |  5|
  |  6|
  |  7|
  |  8|
  |  9|
  | 10|
  +---+
  
  +---+
  |ids|
  +---+
  |  6|
  |  7|
  |  7|
  |  9|
  | 10|
  +---+
  
  +---+
  |ids|
  +---+
  |  1|
  |  3|
  |  7|
  |  8|
  |  9|
  +---+