Use spark in a sbt project in intellij

Question

Here is the build.sbt file:

name := "scalaChartTest"

version := "1.0"

scalaVersion := "2.11.7"

//libraryDependencies += "org.jfree" % "jfreechart" % "1.0.19"
//
//libraryDependencies += "org.scalafx" %% "scalafx" % "8.0.0-R4"
//
//libraryDependencies += "com.github.wookietreiber" %% "scala-chart" % "latest.integration"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.4.1"

And I got an error after refreshing:

15:56:30 SBT project import
         [warn] Multiple dependencies with the same organization/name but different versions. To avoid conflict, pick one version:
         [warn]  * org.scala-lang:scala-compiler:(2.11.0, 2.11.7)
         [warn]  * org.scala-lang:scala-reflect:(2.11.2, 2.11.7)
         [warn]  * jline:jline:(0.9.94, 2.12.1)
         [warn]  * org.scala-lang.modules:scala-parser-combinators_2.11:(1.0.1, 1.0.4)
         [warn]  * org.scala-lang.modules:scala-xml_2.11:(1.0.1, 1.0.4)
         [warn]  * org.slf4j:slf4j-api:(1.6.4, 1.7.10)

What went wrong here?

Answer 1

You have indirect dependencies to the libraries that are mentioned in the warning. There is a conflict, since the version in the indirect dependency is different from what is specified in your sbt-file (in this case probably by the scala version). The conflict is automatically resolved by sbt (chosing either of the versions depending on your configuration). However, the version that is chosen automatically may not be the version you intend to use, hence the warning.

In your case, this probably isn't an issue. Though, if you want to, you can explicitly exclude the indirect dependencies:

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.4.1" excludeAll (
  ExclusionRule(organization = "org.scala-lang"),
  ExclusionRule("jline", "jline"),
  ExclusionRule("org.slf4j", "slf4j-api")
  )