java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration

Question

I keep getting this error.I've included the hadoop commons and the core libs in the classpath but still i keep getting this error.Help would be highly appreciated

Answer 1

Here's how to troubleshoot: Look inside the jar that you're executing to see if that class file is actually there:

jar tvf target/my-jar-with-dependencies.jar | grep hadoop/conf/Configuration.class

If it's not, you need to add it to your classpath or change the way your jar is packaged.

Are you using Maven or some similar build tool? You may have a dependency with a 'scope', which means that it will only be compiled into your jar in certain circumstances.

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>${hadoop.version}</version>
        <scope>provided</scope>
    </dependency>

In this example, the scope tag tells Maven that you're using this dependency for building, but it indicates that the dependency will be provided during runtime, so you'll either need to remove this tag or add the hadoop jar using -cp=/path/to/jar.jar during runtime. Another example of a scope like this is 'test', which indicates that the jar is only needed in the path during unit tests.