Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

DataNode is Not Starting in singlenode hadoop 2.6.0

Tags:

I installed hadoop 2.6.0 in my laptop running Ubuntu 14.04LTS. I successfully started the hadoop daemons by running start-all.sh and I run a WourdCount example successfully, then I tried to run a jar example that didn't work with me so I decide to format using hadoop namenode -format and start all over again but when I start all daemons using start-dfs.sh && start-yarn.sh then jps all daemons runs but not the datanode as shown bellow:

hdferas@feras-Latitude-E4310:/usr/local/hadoop$ jps
12628 NodeManager
12110 NameNode
12533 ResourceManager
13335 Jps
12376 SecondaryNameNode

How to solve that?

like image 674
Firas M. Awaysheh Avatar asked Mar 20 '15 12:03

Firas M. Awaysheh


People also ask

How do I manually start my DataNode?

Datanode daemon should be started manually using $HADOOP_HOME/bin/hadoop-daemon.sh script. Master (NameNode) should correspondingly join the cluster after automatically contacted. New node should be added to the configuration/slaves file in the master server. New node will be identified by script-based commands.

How do I add a DataNode to a Hadoop cluster?

Start the DataNode on New NodeStart the datanode daemon manually using $HADOOP_HOME/bin/hadoop-daemon.sh script. It will automatically contact the master (NameNode) and join the cluster. We should also add the new node to the conf/slaves file in the master server. The script-based commands will recognize the new node.

Why DataNode is shutting down?

DataNode is shutting down reported due to DataNode failed volumes in Pivotal HD.

What happens when DataNode is down?

If Namenode gets down then the whole Hadoop cluster is inaccessible and considered dead. Datanode stores actual data and works as instructed by Namenode. A Hadoop file system can have multiple data nodes but only one active Namenode.


1 Answers

I have faced this issue and it is very easy to solve. Your datanode is not starting because after your namenode and datanode started running you formatted the namenode again. That means you have cleared the metadata from namenode. Now the files which you have stored for running the word count are still in the datanode and datanode has no idea where to send the block reports since you formatted the namenode so it will not start.

Here are the things you need to do to fix it. Stop all the Hadoop services (stop-all.sh) and close any active ssh connections.

cat /usr/local/hadoop/etc/hadoop/hdfs-site.xml

This step is important, see where datanode's data is gettting stored. It is the value associated for datanode.data.dir. For me it is /usr/local/hadoop/hadoop_data/hdfs/datanode. Open your terminal and navigate to above directory and delete the directory named current which will be there under that directory. Make sure you are only deleting the "current" directory.

sudo rm -r /usr/local/hadoop/hadoop_data/hdfs/datanode/current

Now format the namenode and check whether everything is fine.

hadoop namenode -format

say yes if it asks you for anything.

jps

Hope my answer solves the issue. If it doesn't let me know.

Little advice: Don't format your namenode. Without namenode there is no way to reconstruct the data. If your wordcount is not running that is some other problem.

like image 187
Srimanth Avatar answered Oct 23 '22 11:10

Srimanth