Resolving Dependencies in creating JAR through SBT assembly

Question

I want to create a big Jar file. for which am trying to use SBT ASSEMBLY. I installed sbt-assembly from GitHub and this answer. When I ran sbt assembly, I got this error:

java.lang.RuntimeException: deduplicate: different file contents found in the following:
/home/UserName/.ivy2/cache/org.eclipse.jetty.orbit/javax.servlet/orbits/javax.servlet-2.5.0.v201103041518.jar:javax/servlet/SingleThreadModel.class
/home/UserName/.ivy2/cache/org.mortbay.jetty/servlet-api/jars/servlet-api-2.5-20081211.jar:javax/servlet/SingleThreadModel.class

To solve this, I followed User's README page and this is the code he suggests.

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case PathList("org", "apache", xs @ _*) => MergeStrategy.last
    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
    case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
    case PathList("project.clj") => MergeStrategy.last
    case PathList("overview.html") => MergeStrategy.last
    case x => old(x)
  }
}

Even after adding this code, I get the same error mentioned before. Please let me know what I am missing. Any help on this error will be appreciated. Thanks!

Update 2 :

Added the exlusion rule as per the link given,

libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % "0.8.0-incubating","com.codahale" % "jerkson_2.9.1" % "0.5.0","org.skife.com.typesafe.config" % "typesafe-config" % "0.3.0").map(_.exclude("javax", "servlet"))

Update 3:

I can locate the library that is causing the issue.

| +-org.apache.avro:avro-ipc:1.7.4
| | +-io.netty:netty:3.4.0.Final (evicted by: 3.5.4.Final)
| | +-io.netty:netty:3.5.4.Final
...
...
| | +-org.mortbay.jetty:jetty-util:6.1.26
| | +-org.mortbay.jetty:jetty:6.1.26
| | | +-org.mortbay.jetty:jetty-util:6.1.26
| | | +-org.mortbay.jetty:servlet-api:2.5-20081211
| | | 
| | +-org.mortbay.jetty:servlet-api:2.5-20081211
| | +-org.slf4j:slf4j-api:1.7.2
...
... 
| +-org.eclipse.jetty:jetty-server:7.6.8.v20121106
| | +-org.eclipse.jetty.orbit:javax.servlet:2.5.0.v201103041518

Update 4 : Fix

So adding MergeStrategy did solve the problem. Even though I had quite a few dependencies, more than 10, adding MergeStrategy for each one of them individually solved the issue.

Answer 1

I think you are trying to treat the symptoms, but your problem actually isn't assembly: Your project has a library twice in two different versions on the class path (javax.servlet).

If these versions are binary compatible (which I don't know), you would be fine by excluding one of the two occurrences in your build file like so. If they incompatible, you will need to unroll your dependency graph (a good way to do this might be the sbt-dependency-graph plugin) and try to find matching versions.

In any case, it might be useful to (at least transitorily) keep the libraries in the project folder. If you add retrieveManaged in ThisBuild := true to your sbt build file, all the libraries will be found in <project-root>/lib_managed. This allows you to see which jars are actually there.

---

EDIT: Showing the dependency graph:

Add to project/plugins.sbt:

addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.4")

Add to build.sbt:

net.virtualvoid.sbt.graph.Plugin.graphSettings

Then run sbt dependency-graph.