hadoop fs -ls results in "no such file or directory"

Question

I have installed and configured Hadoop 2.5.2 for a 10 node cluster. 1 is acting as masternode and other nodes as slavenodes.

I have problem in executing hadoop fs commands. hadoop fs -ls command is working fine with HDFS URI. It gives message "ls: `.': No such file or directory" when used without HDFS URI

ubuntu@101-master:~$ hadoop fs -ls 15/01/30 17:03:49 WARN util.NativeCodeLoader: Unable to load native-hadoop  ibrary for your platform... using builtin-java classes where applicable ls: `.': No such file or directory ubuntu@101-master:~$  

Whereas, executing the same command with HDFS URI

ubuntu@101-master:~$ hadoop fs -ls hdfs://101-master:50000/ 15/01/30 17:14:31 WARN util.NativeCodeLoader: Unable to load native-hadoop       library for your platform... using builtin-java classes where applicable Found 3 items drwxr-xr-x   - ubuntu supergroup          0 2015-01-28 12:07 hdfs://101-master:50000/hvision-data -rw-r--r--   2 ubuntu supergroup   15512587 2015-01-28 11:50 hdfs://101-master:50000/testimage.seq  drwxr-xr-x   - ubuntu supergroup          0 2015-01-30 17:03 hdfs://101-master:50000/wrodcount-in  ubuntu@101-master:~$  

I am getting exception in MapReduce program due to this behavior. jarlib is referring to the HDFS file location, whereas, I want jarlib to refer to the jar files stored at the local file system on the Hadoop nodes.

Answer 1

The behaviour that you are seeing is expected, let me explain what's going on when you are working with hadoop fs commands.

The command's syntax is this: hadoop fs -ls [path]

By default, when you don't specify [path] for the above command, hadoop expands the path to /home/[username] in hdfs; where [username] gets replaced with linux username who is executing the command.

So, when you execute this command:

ubuntu@xad101-master:~$ hadoop fs -ls 

the reason you are seeing the error is ls: '.': No such file or directory because hadoop is looking for this path /home/ubuntu, it seems like this path doesn't exist in hdfs.

The reason why this command:

ubuntu@101-master:~$ hadoop fs -ls hdfs://101-master:50000/ 

is working because, you have explicitly specified [path] and is the root of the hdfs. You can also do the same using this:

ubuntu@101-master:~$ hadoop fs -ls / 

which automatically gets evaluated to the root of hdfs.

Hope, this clears the behaviour you are seeing while executing hadoop fs -ls command.

Hence, if you want to specify local file system path use file:/// url scheme.