Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

how does hdfs choose a datanode to store

Tags:

hadoop

hdfs

As the title indicates, when a client requests to write a file to the hdfs, how does the HDFS or name node choose which datanode to store the file? Does the hdfs try to store all the blocks of this file in the same node or some node in the same rack if it is too big? Does the hdfs provide any APIs for applications to store the file in a certain datanode as he likes?

like image 595
user1687035 Avatar asked Oct 29 '12 20:10

user1687035


1 Answers

how does the HDFS or name node choose which datanode to store the file?

HDFS has a BlockPlacementPolicyDefault, check the API documentation for more details. It should be possible to extend BlockPlacementPolicy for a custom behavior.

Does the hdfs provide any APIs for applications to store the file in a certain datanode as he likes?

The placement behavior should not be specific to a particular datanode. That's what makes HDFS resilient to failure and also scalable.

like image 62
Praveen Sripati Avatar answered Oct 06 '22 17:10

Praveen Sripati