Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

hadoop hdfs points to file:/// not hdfs://

So I installed Hadoop via Cloudera Manager cdh3u5 on CentOS 5. When I run cmd

hadoop fs -ls /

I expected to see the contents of hdfs://localhost.localdomain:8020/

However, it had returned the contents of file:///

Now, this goes without saying that I can access my hdfs:// through

hadoop fs -ls hdfs://localhost.localdomain:8020/

But when it came to installing other applications such as Accumulo, accumulo would automatically detect Hadoop Filesystem in file:///

Question is, has anyone ran into this issue and how did you resolve it?

I had a look at HDFS thrift server returns content of local FS, not HDFS , which was a similar issue, but did not solve this issue. Also, I do not get this issue with Cloudera Manager cdh4.

like image 893
fair_data Avatar asked Sep 12 '12 15:09

fair_data


People also ask

How copy file from HDFS to local file system?

Hadoop Get command is used to copy files from HDFS to the local file system, use Hadoop fs -get or hdfs dfs -get , on get command, specify the HDFS-file-path where you wanted to copy from and then local-file-path where you wanted a copy to the local file system. Copying files from HDFS file to local file system.

How normal file system is different from HDFS?

Normal file systems have small block size of data. (Around 512 bytes) while HDFS has larger block sizes at around 64 MB) Multiple disks seek for larger files in normal file systems while in HDFS, data is read sequentially after every individual seek.

How data or a file is written into HDFS?

To write a file in HDFS, a client needs to interact with master i.e. namenode (master). Namenode provides the address of the datanodes (slaves) on which client will start writing the data. Client can directly write data on the datanodes, now datanode will create data write pipeline.

What is the most common way to move files from a local file system to HDFS?

Hadoop copyFromLocal command is used to copy the file from your local file system to the HDFS(Hadoop Distributed File System). copyFromLocal command has an optional switch –f which is used to replace the already existing file in the system, means it can be used to update that file.


1 Answers

By default, Hadoop is going to use local mode. You probably need to set fs.default.name to hdfs://localhost.localdomain:8020/ in $HADOOP_HOME/conf/core-site.xml.

To do this, you add this to core-site.xml:

 <property>
  <name>fs.default.name</name>
  <value>hdfs://localhost.localdomain:8020/</value>
</property>

The reason why Accumulo is confused is because it's using the same default configuration to figure out where HDFS is... and it's defaulting to file://

like image 68
Donald Miner Avatar answered Sep 19 '22 12:09

Donald Miner