Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Spark converting a Dataset to RDD

I have a Dataset[String] and need to convert to a RDD[String]. How?

Note: I've recently migrated from spark 1.6 to spark 2.0. Some of my clients were expecting RDD but now Spark gives me Dataset.

like image 830
TakeSoUp Avatar asked Aug 05 '16 19:08

TakeSoUp


2 Answers

As stated in the scala API documentation you can call .rdd on your Dataset :

val myRdd : RDD[String] = ds.rdd
like image 166
cheseaux Avatar answered Oct 12 '22 21:10

cheseaux


Dataset is a strong typed Dataframe, so both Dataset and Dataframe could use .rdd to convert to a RDD.

like image 26
user3215496 Avatar answered Oct 12 '22 20:10

user3215496