Warnings while building Scala/Spark project with SBT

Question

I'm trying to build a Scala/Spark project in IntelliJ Idea with the following build.sbt:

name := "try"  version := "1.0"  scalaVersion := "2.11.8"  val sparkVersion = "2.2.0"  resolvers ++= Seq(   "apache-snapshots" at "http://repository.apache.org/snapshots/" )  libraryDependencies ++= Seq(   "org.apache.spark" %% "spark-core" % sparkVersion,   "org.apache.spark" %% "spark-sql" % sparkVersion,   "org.apache.spark" %% "spark-mllib" % sparkVersion,   "org.apache.spark" %% "spark-streaming" % sparkVersion,   "org.apache.spark" %% "spark-hive" % sparkVersion ) 

and getting a bunch of warnings:

8/6/17 1:29 PM SBT project import                 [warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:                 [warn]  * io.netty:netty:3.9.9.Final is selected over {3.6.2.Final, 3.7.0.Final}                 [warn]      +- org.apache.spark:spark-core_2.11:2.2.0             (depends on 3.9.9.Final)                 [warn]      +- org.apache.zookeeper:zookeeper:3.4.6               (depends on 3.6.2.Final)                 [warn]      +- org.apache.hadoop:hadoop-hdfs:2.6.5                (depends on 3.6.2.Final)                 [warn]  * commons-net:commons-net:2.2 is selected over 3.1                 [warn]      +- org.apache.spark:spark-core_2.11:2.2.0             (depends on 2.2)                 [warn]      +- org.apache.hadoop:hadoop-common:2.6.5              (depends on 3.1)                 [warn]  * com.google.guava:guava:11.0.2 is selected over {12.0.1, 16.0.1}                 [warn]      +- org.apache.hadoop:hadoop-yarn-client:2.6.5         (depends on 11.0.2)                 [warn]      +- org.apache.hadoop:hadoop-yarn-api:2.6.5            (depends on 11.0.2)                 [warn]      +- org.apache.hadoop:hadoop-yarn-common:2.6.5  

I have several, perhaps dumb, questions:

1. Is there a better way to structure build.sbt (add other resolvers e.g.?), so that I can get rid off the warnings?
2. Should I care about warnings at all?

Answer 1

Is there a better way to structure build.sbt (add other resolvers e.g.?), so that I can get rid off the warnings?

One way is to manually tell sbt what dependencies you prefer, for your case:

dependencyOverrides ++= Set(   "io.netty" % "netty" % "3.9.9.Final",   "commons-net" % "commons-net" % "2.2",   "com.google.guava" % "guava" % "11.0.2" ) 

I also recommend reading about conflict management in sbt.

Should I care about warnings at all?

In your case - no, since your conflicts stem from using only spark-related artifacts released under same version. Spark is a project with big userbase and possibility of jar hell introduced due to transitive dependencies is rather low (although, technically not guaranteed).

In general case - maybe. Typically it is ok in most of the cases, but there is a small possibility of a problem that may need careful manual dependency resolution (if this is possible at all). In these cases it is really hard to tell whether there is a problem before you run your app and bump into some problem like missing class, method, mismatching method signature or some reflection-related problem.

Answer 2

In sbt, Spark is normally listed as a Provided dependency, i.e.,

"org.apache.spark" %% "spark-core" % sparkVersion % Provided 

You may be pulling in unnecessary and conflicting recursive dependencies.