Moving data to hdfs using copyFromLocal switch

Question

I don't know what's going on here but I am trying to copy a simple file from a directory in my local filesystem to the directory specified for hdfs.

In my hdfs-site.xml I have specified that the directory for hdfs will be /home/vaibhav/Hadoop/dataNodeHadoopData using the following properties -

<name>dfs.data.dir</name>
<value>/home/vaibhav/Hadoop/dataNodeHadoopData/</value>

and 

<name>dfs.name.dir</name>
<value>/home/vaibhav/Hadoop/dataNodeHadoopData/</value>

I am using the following command -

bin/hadoop dfs -copyFromLocal /home/vaibhav/ml-100k/u.data /home/vaibhav/Hadoop/dataNodeHadoopData

to copy the file u.data from it's local filesystem location to the directory that I specified as Hdfs directory. But when I do this, nothing happens - no error, nothing. And no file gets copied to the hdsf. Am I doing something wrong? Any permissions issue could be there?

Suggestions needed.

I am using pseudo distributed single node mode.

Also, on a related note, I want to ask that in my map reduce program I have set the configuration to point to the inputFilePath as /home/vaibhav/ml-100k/u.data. So would it not automatically copy the file from given location to hdfs ?

Answer 1

I believe dfs.data.dir and dfs.name.dir have to point to two different and existing directories. Furthermore make sure you have formatted the namenode FS after changing the directories in the configuration.

While copying to HDFS you're incorrectly specifying the target. The correct syntax for copying a local file to HDFS is:

bin/hadoop dfs -copyFromLocal <local_FS_filename> <target_on_HDFS>

Example:

bin/hadoop dfs -copyFromLocal /home/vaibhav/ml-100k/u.data my.data

This would create a file my.data in your user's home directory in HDFS. Before copying files to HDFS make sure, you master listing directory contents and directory creation first.