Apache Spark: How do I convert a Spark DataFrame to a RDD with type RDD[(Type1,Type2, ...)]?

Question

For example, suppose I have the DataFrame:

var myDF = sc.parallelize(Seq(("one",1),("two",2),("three",3))).toDF("a", "b")

I can convert it to a RDD[(String, Int)] with a map:

var myRDD = myDF.map(r => (r(0).asInstanceOf[String], r(1).asInstanceOf[Int]))

Is there a better way to do this, maybe using the DF schema?

Answer 1

Using pattern matching over Row:

import org.apache.spark.sql.Row

myDF.map{case Row(a: String, b: Int) => (a, b)}

In Spark 1.6+ you can use Dataset as follows:

myDF.as[(String, Int)].rdd