My code works well in spark-shell:
scala> case class Person(name:String,age:Int)
defined class Person
scala> val person = Seq(Person("ppopo",23)).toDS()
person: org.apache.spark.sql.Dataset[Person] = [name: string, age: int]
scala> person.show()
+-----+---+
| name|age|
+-----+---+
|ppopo| 23|
+-----+---+
but wrong in IDEA:
I have imported all jars in "spark-2.0.0-bin-hadoop2.7/jars/",but still can't find this function.
I find the problem , add a dependency before usering toDS():
val ss = SparkSession.builder().appName("DataSet Test")
.master("local[*]").getOrCreate()
// This import is needed
import ss.implicits._
val simpleDS = Seq(Person("po",12)).toDS()
simpleDS.show()
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With