Meaning of fs.defaultFS property in core-site.xml in hadoop

Question

I am trying to set up hadoop in fully distributed mode, and to some extent I am successful in doing this.

However, I have got some doubt in one of the parameter setting in core-site.xml --> fs.defaultFS

In my set up, I have three nodes as described below:

Node1 -- 192.168.1.2 --> Configured to be Master (Running ResourceManager and NameNode daemons)

Node2 -- 192.168.1.3 --> Configured to be Slave (Running NodeManager and Datanode daemons)

Node3 -- 192.168.1.4 --> Configured to be Slave (Running NodeManager and Datanode daemons)

Now what does property fs.defaultFS mean? For example, if I set it like this:

<property>
   <name>fs.default.name</name>
   <value>hdfs://192.168.1.2:9000/</value>
</property>

I am not able to understand the meaning of hdfs://192.168.1.2:9000. I can figure out that hdfs would mean that we are using hdfs file system, but what does the other parts means?

Does this mean that the host with IP address 192.168.1.2 is running the Namenode at port 9000?

Can anyone help me understand this?

Answer 1

In this code :

<property>
   <name>fs.default.name</name>
   <value>hdfs://192.168.1.2:9000/</value>
</property>

Include fs.defaultFS/fs.default.name in core-site.xml to allow dfs commands without providing full site name in the command. Running hdfs dfs -ls / instead of hdfs dfs -ls hdfs://hdfs/

This is used to specify the default file system and defaults to your local file system that's why it needs be set to a HDFS address. This is important for client configuration as well so your local configuration file should include this element.

Above @Shashank explained very appropriate that :

hdfs://192.168.1.2:9000/. Here 9000 denotes port on which datanode will send heartbeat to namenode. And full address is the machine name that is converted to hostname.

Thanks,

Ankur Singh

Answer 2

<name>fs.default.name</name>.

Here fs denotes file system and default.name denotes namenode

<value>hdfs://192.168.1.2:9000/</value>.

Here 9000 denotes port on which datanode will send heartbeat to namenode. And full address is the machine name that is converted to hostname.

Something important to note about port is that you can give any port greater than 1024 as lesser than that have to give root privileges.