Shading over third party classes

Question

I'm currently facing a problem with deploying an uber-jar to a Spark Streaming application, where there are congruent JARs with different versions which are causing spark to throw run-time exceptions. The library in question is TypeSafe Config.

After attempting many things, my solution was to defer to shading the provided dependency so it won't clash with the JAR provided by Spark at run-time.

Hence, I went to the documentation for sbt-assembly and under shading, I saw the following example:

assemblyShadeRules in assembly := Seq(
      ShadeRule.rename("org.apache.commons.io.**" -> "shadeio.@1")
      .inLibrary("commons-io" % "commons-io" % "2.4", ...).inProject
)

Attempting to shade over com.typesafe.config, I tried applying the following solution to my build.sbt:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.typesafe.config.**" -> "shadeio.@1").inProject
)

I assumed it was supposed to rename any reference to TypeSafe Config in my project. But, this doesn't work. It matches multiple classes in my project and causing them to be removed from the uber jar. I see this when trying to run sbt assembly:

Fully-qualified classname does not match jar entry:
  jar entry: ***/Identifier.class
  class name: **/Identifier.class
Omitting ***/OtherIdentifier.class.
Fully-qualified classname does not match jar entry:
  jar entry: ***\SparkBaseJobRunner$$anonfun$1.class
  class name: ***/SparkBaseJobRunner$$anonfun$1.class

I also attempted using:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.typesafe.config.**" -> "shadeio.@1")
           .inLibrary("com.typesafe" % "config" % "1.3.0")

This did finish the assemblying process of the uber JAR, but didn't have the desired run time effect.

I'm not sure I fully comprehend the effect shading has on my build process with sbt.

How can I shade over references to com.typesafe.config in my project so when I invoke the library at run-time Spark will load my shaded library and avoid the clash caused by versioning?

I'm running sbt-assembly v0.14.1

Answer 1

Turns out this was a bug in sbt-assembly where shading was completely broken on Windows. This caused source files to be removed from the uber JAR, and for tests to fail as the said classes were unavailable.

I created a pull request to fix this. Starting version 0.14.3 of SBT, the shading feature works properly. All you need to do is update to the relevant version in plugins.sbt:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")

In order to shade a specific JAR in your project, you do the following:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.typesafe.config.**" -> "my_conf.@1")
    .inLibrary("com.typesafe" % "config" % "1.3.0")
    .inProject
)

This will rename the com.typesafe.config assembly to be packaged inside my_conf. You can then view this using jar -tf on your assembly (omitted irrelevant parts for brevity):

***> jar -tf myassembly.jar
my_conf/
my_conf/impl/
my_conf/parser/

Edit

I wrote a blog post describing the issue and the process that led to it for anyone interested in a more in-depth explanation.