JAR HELL in IDE (intelliJ), mvn-test and mvn-package: how to unify dependency relocation in all settings?

Question

Simple case for me: I'm using 2 very popular but conflicting scala libraries: Spark SQL & JSON4s. Each depends on different versions of the Jackson XML parser.

Since nobody except my own program is using JSON4s, a simple solution is to migrate the packages org.json4s.jackson and com.fasterxml.jackson to new locations.

However, the maven-shade plugin can only do this in the packaging process, since all tests and IDE runs happen before that. This still cause all tests to fail, regardless of whether to be run in mvn-test or scala test in the IDE.

Is there a plugin that can collectively manage package relocation policies for all three cases: JAR-packaging/mvn-test/IDE-run? I've been searching the Internet all day and can't find an answer.

Answer 1

We solved the issue by recompiling (and patching) to align the libraries. Anything shade related is going to favor one version over the other, which means that what you might fix in one library you might break in another.

And for those comments from the OSGi camp, yes, that's what OSGi was meant to fix, but it doesn't work out so well in a Spark context :)

Have you considered downgrading one of the two versions to align resources? Often one of the libraries manages to release first, leaving the second one behind by just a bit... Sometimes the answer can be found in running a consistent yet older version that aligns with the "last to be released" project.