Maven build [WARNING] we have a duplicate class

Question

Anybody has any idea what happened to my maven build? I am getting a lot of duplicate warnings.

[WARNING] We have a duplicate org/apache/commons/logging/impl/LogFactoryImpl$1.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar [WARNING] We have a duplicate org/apache/commons/logging/impl/LogFactoryImpl.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar [WARNING] We have a duplicate org/apache/commons/logging/impl/NoOpLog.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar [WARNING] We have a duplicate org/apache/commons/logging/impl/SimpleLog$1.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar [WARNING] We have a duplicate org/apache/commons/logging/impl/SimpleLog.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar [WARNING] We have a duplicate org/apache/commons/logging/impl/Jdk14Logger.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar 

I've looked into my local m2 repo, I have two classes there in commons-logging-api jar, LogFactoryImpl.class and LogFactoryImpl$1.class. Same as all the classes mentioned in the warnings.

One thing to mention is that I am using shade plugin in my pom.xml.

        <plugin>             <groupId>org.apache.maven.plugins</groupId>             <artifactId>maven-shade-plugin</artifactId>             <version>1.4</version>             <configuration>                 <createDependencyReducedPom>true</createDependencyReducedPom>                 <filters>                     <filter>                         <artifact>*:*</artifact>                         <excludes>                             <exclude>META-INF/*.SF</exclude>                             <exclude>META-INF/*.DSA</exclude>                             <exclude>META-INF/*.RSA</exclude>                         </excludes>                     </filter>                 </filters>             </configuration>             <executions>                 <execution>                     <phase>package</phase>                     <goals>                         <goal>shade</goal>                     </goals>                     <configuration>                         <transformers>                             <transformer                                 implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />                             <transformer                                 implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">                                 <mainClass>com.~~~~black out my own main class here~~~~~</mainClass>                             </transformer>                         </transformers>                     </configuration>                 </execution>             </executions>         </plugin> 

I noticed that the dependency tree looks like as below

[INFO] +- org.apache.cxf:cxf-bundle-jaxrs:jar:2.5.1:compile [INFO] |  \- commons-logging:commons-logging:jar:1.1.1:compile [INFO] \- org.apache.hadoop.hive:hive-jdbc:jar:0.7.1-cdh3u3:compile [INFO]    \- org.apache.hadoop.hive:hive-common:jar:0.7.1-cdh3u3:compile [INFO]       \- commons-logging:commons-logging-api:jar:1.0.4:compile 

and commons-logging.jar and commons-logging-api.jar both have org/apache/commons/logging/LogFactory.class.

somehow Shad plugin is trying to squeeze them in to a big fat jar at the end. then the warning is showing up. It's been said this is ignorable warning. But I am a bit worried, How does the application know what is the exact class should be used if there are two duplicated class with the same name?

Answer 1

You might also have run into a limitation of maven-shader-plugin. It replaces the default jar artifact (created by maven-jar-plugin). This works fine on a clean build, but on a rebuild where the jar is not regenerated, the shader runs again on the jar it created last time, which already contains copies of all the class dependencies. That produces a lot of warnings about duplicates.

This issue is still unaddressed as of maven-shader-plugin 2.0: https://issues.apache.org/jira/browse/MSHADE-126

One workaround is to add the maven-jar-plugin explicitly to your pom.xml and add the configuration setting <forceCreation>true</forceCreation>.