Scala SBT Assembly cannot merge due to de-duplication error in StaticLoggerBinder.class

Question

My problem is I can no longer use the sbt-assembly plugin because some kind of dependency merge problem creeped in, between a couple people working on this project.

The problem when I run 'sbt assembly' :

[error] 3 errors were encountered during merge java.lang.RuntimeException: deduplicate: different file contents found in the following: /Users/aris.vlasakakis/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.2.jar:org/slf4j/impl/StaticLoggerBinder.class /Users/aris.vlasakakis/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.6.1.jar:org/slf4j/impl/StaticLoggerBinder.class deduplicate: different file contents found in the following: /Users/aris.vlasakakis/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.2.jar:org/slf4j/impl/StaticMDCBinder.class /Users/aris.vlasakakis/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.6.1.jar:org/slf4j/impl/StaticMDCBinder.class deduplicate: different file contents found in the following: /Users/aris.vlasakakis/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.2.jar:org/slf4j/impl/StaticMarkerBinder.class /Users/aris.vlasakakis/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.6.1.jar:org/slf4j/impl/StaticMarkerBinder.class at sbtassembly.Assembly$.applyStrategies(Assembly.scala:140) at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25) at sbtassembly.Assembly$.x$1$1(Assembly.scala:23) at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23) at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23) at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67) at sbtassembly.Assembly$.inputs$1(Assembly.scala:57)

... and so on

I am using SBT-assembly 0.13.0, and here is the build.sbt

name := "metamorphosis"

version := "0.10.0"

scalaVersion := "2.10.4"

lazy val common = RootProject(file("../"))

val main = Project(id = "bridge", base = file(".")).dependsOn(common)

//excludeDependencies += "org.apache.logging.log4j"

resolvers ++= Seq(
//  "Akka Snapshot Repository" at "http://repo.akka.io/snapshots/",
  "Typesafe Releases" at "http://repo.typesafe.com/typesafe/releases/",
  "Sonatype OSS Releases"  at "http://oss.sonatype.org/content/repositories/releases/"
)

libraryDependencies ++= Seq(
  "com.github.nscala-time" %% "nscala-time" % "1.8.0",
  "com.sclasen" %% "akka-kafka" % "0.0.10" % "compile",
  "com.typesafe.akka" %% "akka-actor" % "2.3.2",
  "org.codehaus.groovy" % "groovy" % "2.3.7",
  "com.typesafe.scala-logging" %% "scala-logging-slf4j" % "2.1.2",
  "ch.qos.logback" % "logback-classic" % "1.1.2",
  "ch.qos.logback.contrib" % "logback-json-classic" % "0.1.2",
  "ch.qos.logback.contrib" % "logback-jackson" % "0.1.2",
  "org.slf4j" % "slf4j-api" % "1.7.7",
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.2.2",
  "org.clapper" %% "argot" % "1.0.3",
  "com.typesafe" % "config" % "1.2.1",
  "net.ceedubs" %% "ficus" % "1.0.1",
  "com.typesafe.play" %% "anorm" % "2.3.6",
  "org.json4s" %% "json4s-native" % "3.2.10",
  "org.json4s" %% "json4s-jackson" % "3.2.10",
  "com.github.tototoshi" %% "scala-csv" % "1.1.2",
  "org.scalatest" %% "scalatest" % "2.2.2",
  "junit" % "junit" % "4.11",
  "org.apache.kafka" %% "kafka" % "0.8.1.1"
    exclude("javax.jms", "jms")
    exclude("com.sun.jdmk", "jmxtools")
    exclude("com.sun.jmx", "jmxri")
    exclude("org.slf4j", "slf4j-simple")
)

mainClass in assembly := Some("com.company.kafka.agent.MetamorphosisActor")

Any clues on how I can even figure out how to fix this merge problem?

Thank you!

Answer 1

Previous answer gives all the required feedback, but does not actually give any solution or steps to try out.

My error:

[error] (project/*:assembly) deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.2.3.jar:org/slf4j/impl/StaticLoggerBinder.class
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.10.jar:org/slf4j/impl/StaticLoggerBinder.class
[error] deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.2.3.jar:org/slf4j/impl/StaticMDCBinder.class
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.10.jar:org/slf4j/impl/StaticMDCBinder.class
[error] deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.2.3.jar:org/slf4j/impl/StaticMarkerBinder.class
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.10.jar:org/slf4j/impl/StaticMarkerBinder.class

Recommended way

track down conflicting dependency. For example, I do not want those classes:

org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.10.jar:org/slf4j/impl/StaticLoggerBinder.class
org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.10.jar:org/slf4j/impl/StaticMDCBinder.class
org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.10.jar:org/slf4j/impl/StaticMarkerBinder.class

Call sbt (and make sure to use sbt-dependency-graph plugin):

whatDependsOn org.slf4j slf4j-log4j12 1.7.10

Returns a list:

[info] org.slf4j:slf4j-log4j12:1.7.10
[info]   +-org.apache.hadoop:hadoop-auth:2.8.0
[info]   | +-org.apache.hadoop:hadoop-common:2.8.0
[info]   |   +-com.github.atais:test_2.11:0.0.3 [S]
[info]   |   
[info]   +-org.apache.hadoop:hadoop-common:2.8.0
[info]   | +com.github.atais:test_2.11:0.0.3 [S]
[info]   | 
[info]   +-org.apache.zookeeper:zookeeper:3.4.6
[info]     +-org.apache.curator:curator-client:2.7.1
[info]     | +-org.apache.curator:curator-framework:2.7.1
[info]     | | +-org.apache.curator:curator-recipes:2.7.1
[info]     | | | +-org.apache.hadoop:hadoop-common:2.8.0
[info]     | | |   +-com.github.atais:test_2.11:0.0.3 [S]
[info]     | | |   
[info]     | | +-org.apache.hadoop:hadoop-auth:2.8.0
[info]     | |   +-org.apache.hadoop:hadoop-common:2.8.0
[info]     | |     +-com.github.atais:test_2.11:0.0.3 [S]
[info]     | |     
[info]     | +-org.apache.hadoop:hadoop-common:2.8.0
[info]     |   +-com.github.atais:test_2.11:0.0.3 [S]
[info]     |   
[info]     +-org.apache.curator:curator-framework:2.7.1
[info]     | +-org.apache.curator:curator-recipes:2.7.1
[info]     | | +-org.apache.hadoop:hadoop-common:2.8.0
[info]     | |   +-com.github.atais:test_2.11:0.0.3 [S]
[info]     | |   
[info]     | +-org.apache.hadoop:hadoop-auth:2.8.0
[info]     |   +-org.apache.hadoop:hadoop-common:2.8.0
[info]     |     +-com.github.atais:test_2.11:0.0.3 [S]
[info]     |     
[info]     +-org.apache.curator:curator-recipes:2.7.1
[info]     | +-org.apache.hadoop:hadoop-common:2.8.0
[info]     |   +-com.github.atais:test_2.11:0.0.3 [S]
[info]     |   
[info]     +-org.apache.hadoop:hadoop-auth:2.8.0
[info]     | +-org.apache.hadoop:hadoop-common:2.8.0
[info]     |   +-com.github.atais:test_2.11:0.0.3 [S]
[info]     |   
[info]     +-org.apache.hadoop:hadoop-common:2.8.0
[info]       +-com.github.atais:test_2.11:0.0.3 [S]

I need to track down hadoop-common dependency since it is the one that connects com.github.atais:test_2.11:0.0.3 to the unwanted org.slf4j:slf4j-log4j12:1.7.10

And modify it:

libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.8.1" exclude("org.slf4j", "slf4j-log4j12")

Easy way

simply add the merging strategy flag for conflicting paths:

assemblyMergeStrategy in assembly := {
  ...
  case PathList("org", "slf4j", xs@_*) => MergeStrategy.first
  case x => (assemblyMergeStrategy in assembly).value(x)
}

Answer 2

The issue is that more than one dependency brings in slf4j transitive dependency which causes a conflict. The best way is to exclude that transitive dependency instead of using merging strategy of SBT assembly to resolve the conflict.

First you need to visualize where slf4j comes from (for all top level dependencies). You can use a tool like this sbt-dependency-graph to get a visual diagram of all dependencies. Alternatively you can explore SBT settings from SBT command prompt or use a maven site like this one or this one.

Once you locate duplicate dependencies use exclude the same way as you use in your build file. Leave only single slf4j transitive dependency. You can also exclude all slf4j transitive dependencies (see here how) and manually add that slf4j dependency at top level.

Note that if your dependencies require multiple different versions of slf4j it might be problematic to pick a single version that works for all cases. Depends on binary compatibility.