sbt assembly error:deduplicate: different file contents found in io.netty.versions.properties

Question

I have added the following jar to build.sbt file as follows:

"com.amazonaws" % "aws-java-sdk" % "1.11.492"

Post this ,during merge , I am getting the following error :

 [error] 1 error was encountered during merge
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/home/jenkins-slave/.ivy2/cache/io.netty/netty-codec-http/jars/netty-codec-http-4.1.17.Final.jar:META-INF/io.netty.versions.properties
/home/jenkins-slave/.ivy2/cache/io.netty/netty-handler/jars/netty-handler-4.1.17.Final.jar:META-INF/io.netty.versions.properties
/home/jenkins-slave/.ivy2/cache/io.netty/netty-codec/jars/netty-codec-4.1.17.Final.jar:META-INF/io.netty.versions.properties
/home/jenkins-slave/.ivy2/cache/io.netty/netty-transport/jars/netty-transport-4.1.17.Final.jar:META-INF/io.netty.versions.properties
/home/jenkins-slave/.ivy2/cache/io.netty/netty-resolver/jars/netty-resolver-4.1.17.Final.jar:META-INF/io.netty.versions.properties
/home/jenkins-slave/.ivy2/cache/io.netty/netty-buffer/jars/netty-buffer-4.1.17.Final.jar:META-INF/io.netty.versions.properties
/home/jenkins-slave/.ivy2/cache/io.netty/netty-common/jars/netty-common-4.1.17.Final.jar:META-INF/io.netty.versions.properties
    at sbtassembly.Assembly$.applyStrategies(Assembly.scala:143)
    at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
    at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
    at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
    at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)...........

I have tried many workarounds provided for this like:

1) added this line in assemblyMergeStrategy in build.sbt:case PathList("io", "netty", xs @ _*) => MergeStrategy.discard(tried with .last and .first)

2)added this line in assemblyMergeStrategy in build.sbt:case "META-INF\\io.netty.versions.properties" =>MergeStrategy.first(tried with .last and .discard)

3)added SBT exclusion rules for the errored out netty jars in excludedDependencies like below:

    excludeDependencies ++= Seq(
      SbtExclusionRule("io.netty", "netty-codec-http"),
      SbtExclusionRule("io.netty", "netty-codec"),
      SbtExclusionRule("io.netty", "netty-handler"),
      SbtExclusionRule("io.netty", "netty-transport"),
      SbtExclusionRule("io.netty", "netty-resolver"),
      SbtExclusionRule("io.netty", "netty-buffer"),
      SbtExclusionRule("io.netty", "netty-common")
    )

and many such variations of the above.None of these solutions are working.

plugins.sbt looks like below:

addSbtPlugin("io.spray" % "sbt-revolver" % "0.7.2")

addSbtPlugin("com.eed3si9n" % "sbt-buildinfo" % "0.5.0")

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.9")

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.0.1")

addSbtPlugin("com.scalapenos" % "sbt-prompt" % "0.2.1")

addSbtPlugin("com.timushev.sbt" % "sbt-updates" % "0.1.10")

addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.8.0")

addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")

Kindly advise.

Answer 1

You could try below:

assemblyMergeStrategy in assembly := {
      case x if x.contains("io.netty.versions.properties") => MergeStrategy.discard
      case x =>
        val oldStrategy = (assemblyMergeStrategy in assembly).value
        oldStrategy(x)
}

or in worst case

case x if x.contains("versions.properties") => MergeStrategy.discard

Answer 2

If you check the contents of the file you can see that every jar has different values in it, so to be correct in the assembly you have concat these files to preserve all the properties in them. Use case "META-INF/io.netty.versions.properties" => MergeStrategy.concat it the strategy.

Answer 3

I've run into the same issue on windows, and found the solution should be:

case PathList("META-INF", "io.netty.versions.properties") => MergeStrategy.first

or putting it in the assemblyMergeStrategy settings, it will be:

assembly / assemblyMergeStrategy := {
  case PathList("META-INF", "io.netty.versions.properties") => MergeStrategy.first
  case x =>
    val oldStrategy = (assembly / assemblyMergeStrategy).value
    oldStrategy(x)
}

---

I think the reasons why the workarounds mentioned in the question doesn't work is because:

1. the io.netty.versions.properties is a file under META-INF, instead of a file named versions.properties under io/netty. So this one doesn't matched the intended file:

case PathList("io", "netty", xs @ _*) => MergeStrategy.discard

2. I think the reason why the second one doesn't work is because we need to use PathList to match against a path under some folders, instead of using slashes like it shows in the question(with a backslash):

case "META-INF\\io.netty.versions.properties" =>MergeStrategy.first

When I first ran into this issue, I was using the forward slash, which doesn't work either:

case "META-INF/io.netty.versions.properties" =>MergeStrategy.first

3. I think the netty dependencies are required for the aws-java-sdk to work. So excluding them from the dependencies will get us ClassNotFoundException. Which will not solve the problem.