Menu
I am a java programmer, learning Hadoop. I read that the Name node in HDFS stores its information into two files namely fsImage & editLog. In case of start up it reads this data from the disk & performs checkpoint operation.
But at many places I also read that Name Node stores the data in RAM & that is why apache recommends a machine with high RAM for Name Node server.
Please enlighten me on this. What data does it store in RAM & where does it store fsImage and edit Log ?
Sorry if I asked anything obvious.
660
located in your hadoop conf folder. In my system it is located at usr/lib/hadoop/conf which is hadoop installed directory. In this code /var/lib/hadoop-0.20/cache/ is the location of fsimage, fstime and edits log.
The FsImage is stored as a file in the NameNode's local file system too. The NameNode keeps an image of the entire file system namespace and file Blockmap in memory.
When we are starting namenode, latest FsImage file is loaded into "in-memory" and at the same time, EditLog file is also loaded into memory if FsImage file does not contain up to date information. Namenode stores metadata in "in-memory" in order to serve the multiple client request(s) as fast as possible.
Namenode stored metadata in “in-memory” in order to serve the multiple client request(s) as fast as possible.
When namenode starts, it loads fsimage from persistent storage(disk) it's location specified by the property dfs.name.dir (hadoop-1.x) or dfs.namenode.name.dir (hadoop-2.x) in hdfs-site.xml. Fsimage is loaded into main memory. Also as you asked during namenode starting it performs check point operation. Namenode keeps the Fsimage in RAM inorder to serve requests fast.
Apart from initial checkpoint, subsequent checkpoints can be controlled by tuning the following parameters in hdfs-site.xml.
dfs.namenode.checkpoint.period # in second 3600 Secs by default
dfs.namenode.checkpoint.txns # No of namenode transactions
Let me first answer
What data does it store in RAM & where does it store fsImage and edit Log ?
In RAM -- file to block and block to data node mapping. In persistent storage (includes both edit log and fsimage) -- file related metadata (permissions, name and so on)
Regarding the storage location of the fsimage and editlog @mashuai's answer is spot on.
For a more detailed discussion you can read up on this
125
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
