Hadoop namenode metadata

Question

I am a bit confused by the Hadoop architecture.

1. What kind of file metadata is stored in Hadoop Namenode? From Hadoop wiki, it says Namenode stores the entire system namespace. Does information like last modified time, created time, file size, owner, permissions and etc stored in Namenode?

2. Does datanode store any metadata information?

3. There is only one Namenode, can the metadata data exceed the server's limit?

4. If a user wants to download a file from Hadoop, does he have to download it from the Namenode? I found the below architecure picture from web, it shows a client can direct write data to datanode? Is it true?

Thanks!!!!!!!

Answer 1

I think the following explanation can help you to better understand the HDFS architecture. You can consider Name node to be like FAT (file allocation table) + Directory data and Data nodes to be dumb block devices. When you want to read the file from the regular file system, you should go to Directory, then go to FAT, get locations of all relevant blocks and read them. The same happens with HDFS. When you want to read the file, you go to the Namenode, get the list blocks the given file have. This information about blocks will contain list of datanodes where this information sitting. After it you go to the datanode and get relevant blocks from them.

Answer 2

1. The fsimage on the name node is in a binary format. Use the "Offline Image Viewer" to dump the fsimage in a human-readable format. The output of this tool can be further analyzed with pig or some other tool to get more meaningful data.

http://hadoop.apache.org/hdfs/docs/r0.21.0/hdfs_imageviewer.html