Manage multiple clusters in Hadoop OR Distributed Computing Framework

Question

I have five computers networked together. Among them one is master computer and another four are slave computers.

Each slave computer has its own set of data (a very big integer matrix). I want to run four different clustering programs in four different slaves. Then, take the results back into the master computer for further processing (such as visualization).

I initially thought to use Hadoop. But, I cannot find any nice way to convert the above problem (specifically the output results) into the Map Reduce framework.

Is there any nice open-source distributed computing framework by using which I can perform the above task easily?

Thanks in advance.

Answer 1

You should used YARN for manage multiple clusters or resources

YARN is the prerequisite for Enterprise Hadoop, providing resource management and a central platform to deliver consistent operations, security, and data governance tools across Hadoop clusters.

Reference