object DataFrame is not a member of package org.apache.spark.sql

Question

I import org.apache.spark.sql.DataFrame in my scala file, than use sbt to compile, the error was object DataFrame is not a member of package org.apache.spark.sql

Searched for some solutions in the Internet, it seems that the problem is spark version is too old. But I am using the newest version(2.1.1), so it is weird.

In REPL, when I import org.apache.spark.sql.DataFrame, there is no error.

My function is like this:

def test(df: DataFrame): Unit={
    ....
}

When I define this function in REPL, it is fine, but when I compile it using sbt, the error is not found: type DataFrame.

My build.sbt:

name := "Hello"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"

Anyone can help me to fix this problem? Thanks.

Answer 1

You need both spark-core and spark-sql to work with Dataframe

libraryDependencies ++= Seq(
// https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11
  "org.apache.spark" %% "spark-core" % "2.1.1",
// https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.11
  "org.apache.spark" %% "spark-sql" % "2.1.1"
) 

Hope this helps!