How to access files in Hadoop HDFS?

Question

I have a .jar file (containing a Java project that I want to modify) in my Hadoop HDFS that I want to open in Eclipse.

When I type hdfs dfs -ls /user/... I can see that the .jar file is there - however, when I open up Eclipse and try to import it I just can't seem to find it anywhere. I do see a hadoop/hdfs folder in my File System which takes me to 2 folders; namenode and namesecondary - none of these have the file that I'm looking for.

Any ideas? I have been stuck on this for a while. Thanks in advance for any help.

Answer 1

As HDFS is virtual storage it is spanned across the cluster so you can see only the metadata in your File system you can't see the actual data.

Try downloading the jar file from HDFS to your Local File system and do the required modifications.

Access the HDFS using its web UI. Open your Browser and type localhost:50070 You can see the web UI of HDFS move to utilities tab which is on the right side and click on Browse the File system, you can see the list of files which are in your HDFS.

Follow the below steps to download the file to your local file system.

Open Browser-->localhost:50070-->Utilities-->Browse the file system-->Open your required file directory-->Click on the file(a pop up will open)-->Click on download

The file will be downloaded into your local File System and you can do your required modifications.

Answer 2

HDFS filesystem and local filesystem are both different.

You can copy the jar file from hdfs filesystem to your preferred location in your local filesytem by using this command:

bin/hadoop fs -copyToLocal locationOfFileInHDFS locationWhereYouWantToCopyFileInYourFileSystem

For example

bin/hadoop fs -copyToLocal file.jar /home/user/file.jar

I hope this helps you.

Answer 3

1) Get the file from HDFS to your local system

bin/hadoop fs -get /hdfs/source/path /localfs/destination/path

2) you can manage it in this way:

New Java Project -> Java settings -> Source -> Link source (Source folder).