Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why can't IDEA find toDS() and toDF() functions?

My code works well in spark-shell:

scala> case class Person(name:String,age:Int)
defined class Person

scala> val person = Seq(Person("ppopo",23)).toDS()
person: org.apache.spark.sql.Dataset[Person] = [name: string, age: int]

scala> person.show()
+-----+---+
| name|age|
+-----+---+
|ppopo| 23|
+-----+---+

but wrong in IDEA:

enter image description here
I have imported all jars in "spark-2.0.0-bin-hadoop2.7/jars/",but still can't find this function.

like image 303
Pi Pi Avatar asked Aug 16 '16 10:08

Pi Pi


1 Answers

I find the problem , add a dependency before usering toDS():

val ss = SparkSession.builder().appName("DataSet Test")
  .master("local[*]").getOrCreate()

// This import is needed
import ss.implicits._
val simpleDS = Seq(Person("po",12)).toDS()
simpleDS.show()
like image 181
Pi Pi Avatar answered Sep 28 '22 01:09

Pi Pi