Hadoop NoSuchMethodError apache.commons.cli

Question

I'm using hadoop-2.7.2 and I did a MapReduceJob with IntelliJ. In my job, I'm using apache.commons.cli-1.3.1 and I put the lib in the jar.

When I use the MapReduceJob on my Hadoop cluster I have a NoSuchMethodError:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;

I don't understand because the method exist in the class Option and the class Option is extracted from the commons-cli.jar to my application jar. Moreover, I don't have this issue with my others libraries.

Thank you for your time.

Answer 1

We were able to fix this error using maven class relocations. If you are using the shade plugin to build your jar add following to pom.xml under appropriate section:

<!-- necessary to fix NoSuchMethodError: org.apache.commons.cli.Option.builder -->
                            <relocations>
                                <relocation>
                                    <pattern>org.apache.commons.cli</pattern>
                                    <shadedPattern>org.shaded.commons.cli</shadedPattern>
                                </relocation>
                            </relocations>

Also an explicit reference to v1.3+ of commons-cli needs to be added at the TOP of the dependencies section before any dependency that may have a transitive reference to an older version of commons-cli.

Answer 2

The problem seems to be related to how the classloader is loading the classes. Because the static Builder class was in common-cli 1.4, while some of the hadoop dependencies were still referring to older version - the issue occurred.

In my case, the issue resolved by changing the sequence of jar file addition into the classpath in the shell script responsible for setting up environment before program execution. Earlier, I was adding the jar into classpath like

CLASSPATH=<Hadoop Jars>:<Common CLI jar>:$CLASSPATH

is changed to

CLASSPATH=<Common CLI jar>:<Hadoop Jars>:$CLASSPATH

and it fixed the problem.

Answer 3

We solved this issue with the next gradle configuration:

 compile('org.apache.parquet:parquet-tools:1.9.0'){
  exclude module:"commons-cli"
 }