Adding Spark "Library" to a Scala project

Question

I'm going through a book and the author left out the part of actually providing the project with the Scala libraries for Spark. So things like

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

Are not working (failing on the apache).

I'm new to Scala (and JVM languages in general) and am not even sure what to search for. Should I be looking for jars? How would I add them? I'm using intelliJ if that helps with the explanation.

I do have spark running so i think it is the the "client" spark library for scala that I need to install.

So using the answer below I'm much closer now (using sbt now) but I have the following error:

build.sbt is now:

name := "gettingThingsRunning"

version := "1.0"

scalaVersion := "2.12.1"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "1.5.1",
  "org.apache.spark" %% "spark-sql" % "1.5.1" )

Answer 1

You need to use a dependency manager like Maven or sbt.

With sbt, you can add a build.sbt file in your project root folder with something like:

name := "Simple Project"

version := "1.0"

scalaVersion := "2.11.7"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.2"

Then you can use IntelliJ's Scala plug-in to "refresh" your project and re-index the dependencies.

Edit: Assuming you created your project using the SBT option, you can use the SBT functionality found in the sidebar for refreshing your dependencies:

Regarding Spark, I recommend reading the Spark quick start page, particularly the section called "Self-contained applications".

For IntelliJ and SBT projects, I believe you would profit from reading the following page:

https://www.jetbrains.com/help/idea/2016.3/getting-started-with-sbt.html

Answer 2

Change your scalaVersion to 2.11.8. Look at this for details.