Ordering an RDD[String]

Question

Consider

val animals = List("penguin","ferret","cat").toSeq
val rdd = sc.makeRDD(animals, 1) 

I would like to order this RDD. I'm new to Scala and a little confused about how this is to be done.

Answer 1

RDD documentation can be found here. Look at sortBy:

sortBy[K](
  f: (T) ⇒ K, 
  ascending: Boolean = true, 
  numPartitions: Int = this.partitions.size
)

The K is the type of the snippet of the RDD you are sorting by. f is a function, which you can either define elsewhere with def and pass it by name or you can create one anonymously in line (which is more scala-like). ascending and numPartitions should be self explanatory.

So given all this, try:

rdd.sortBy[String]({animal => animal})

Then try this:

rdd.sortBy[String]({animal => animal}, false)

And then this one, which sorts the RDD by the number of letters "e" in the name of the animal, from most to least:

rdd.sortBy[Int]({a => a.split("").filter(char => char == "e").size}, false)

It should be noted that the original rdd isn't sorted -- a new, sorted RDD is returned by the operation.