Trouble building a simple SparkSQL application

Question

This is a pretty noob question.

I'm trying to learn about SparkSQL. I've been following the example described here: http://spark.apache.org/docs/1.0.0/sql-programming-guide.html

Everything works fine in the Spark-shell, but when I try to use sbt to build a batch version, I get the following error message: object sql is not a member of package org.apache.spark

Unfortunately, I'm rather new to sbt, so I don't know how to correct this problem. I suspect that I need to include additional dependencies, but I can't figure out how.

Here is the code I'm trying to compile:

/* TestApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

case class Record(k: Int, v: String)

object TestApp {
 def main(args: Array[String]) {
   val conf = new SparkConf().setAppName("Simple Application")
   val sc = new SparkContext(conf)
   val sqlContext = new org.apache.spark.sql.SQLContext(sc)
   import sqlContext._
   val data = sc.parallelize(1 to 100000)
   val records = data.map(i => new Record(i, "value = "+i))
   val table = createSchemaRDD(records, Record)
   println(">>> " + table.count)
 }
}

The error is flagged on the line where I try to create a SQLContext.

Here is the content of the sbt file:

name := "Test Project"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.0.0"

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

Thanks for the help.

Answer 1

As is often the case, the act of asking the question helped me figure out the answer. The answer is to add the following line in the sbt file.

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0"

I also realized there is an additional problem in the little program above. There are too many arguments in the call to createSchemaRDD. That line should read as follows:

val table = createSchemaRDD(records)