How to run a jar file in hadoop?

Question

I have created a jar file using the java file from this blog using following statements

javac -classpath /usr/local/hadoop/hadoop-core-1.0.3.jar -d /home/hduser/dir Dictionary.java

/usr/lib/jvm/jdk1.7.0_07/bin/jar cf Dictionary.jar /home/hduser/dir

Now i have tried running this jar in hadoop by hit and trial of various commands

1hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop jar Dictionary.jar

Output:

Warning: $HADOOP_HOME is deprecated.

RunJar jarFile [mainClass] args...  

2.hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop jar Dictionary.jar Dictionary

Output:

Warning: $HADOOP_HOME is deprecated.

Exception in thread "main" java.lang.ClassNotFoundException: Dictionary
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)

How can i run the jar in hadoop? I have the right DFS Locations as per needed by my program.

Answer 1

I was able to reproduce your problem. The problem is where you are creating the jar.

Basically, the directory that you are packaging into the jar is confusing the jar file in locating the main class file. Instead if you try doing :

/usr/lib/jvm/jdk1.7.0_07/bin/jar cf Dictionary.jar /home/hduser/dir/Dictionary.class

i.e. package the class file specifically into the jar and then run:

/usr/local/hadoop/bin/hadoop jar Dictionary.jar Dictionary

It just works fine provided that you have a main function in your class called Dictionary.

The problem is when you package a full directory inside a jar then the jar also needs to be aware of the directory structure to locate the class file. For this, we need to have a well defined package hierarchy to define the class location. So, when you are packaging /home/hduser/dir/ into the jar, the jar is not aware of the location of the class file which is located deep inside this directory structure. For this you need to add a package name to your .java file according to the directory structure , for example home.hduser.dir and while running the hadoop jar command specify the class name with the package structure, for example home.hduser.dir.Dictionary.