running Hadoop software on office computers (when they are idle)

Question

Is there a project which helps setup a Hadoop cluster on office desktops, when they are idle?

I'd like to experiment with Hadoop/MR/hbase but don't have acces to 5-10 computers. The computers at work are idle after hours and are connected to each other through a very high speed connection. What's more, data on these computers stays within our network so there is no privacy issue.

In order for this to work I need a fairly light weight monitor running on each machine. When the computer has been idle for X hours, it will join the cluster. If the user logs on, it has to drop out of the cluster and return all CPU/memory back.

Does something like this exist?

Answer 1

You can use task scheduler to detect idle state and then start/stop a hadoop vm with virtual box or vmplayer. Or you can write a powershell script that does start stop based on resource usage.

Answer 2

Hadoop is not a computation grid it is a more a data grid (see slide 9 in this presentation). The point is that with hadoop that data is spread over the cluster and thus the data has to be stored on the computers. The time it would take to copy the data over/remove it when they're not idle would probably not be worth it - you'd be better off using hadoop in the cloud (amazon,Azure etc.)