What is the difference between Rack-local map tasks and Data-local map tasks?

Question

When I run "hadoop job -status xxx",Output the following some list.

Rack-local map tasks=124
Data-local map tasks=6

What is the difference between Rack-local map tasks and Data-local map tasks?

Answer 1

In a data-local task, nothing needs to be copied. That's because the block is physically on the same server like the computation.

The next tier is the rack-local task, here the data must be copied, because there is no local copy of the desired block available. Note that rack-local does only copy within the rack-local switching of the network.

There is also the worst case, where the data isn't available local, nor on the same rack. So this must be copied over two switches to the hosts where the computation runs. I don't know if there is a counter for that, but basically this must be #all tasks - #data-local tasks - #rack-local tasks.

Answer 2

I would point out that providing gigabit (or faster) network between computers within the same rack is much cheaper that for bigger number of computers.
The root cause is the fact that ethernet switches are not scalable and we can not have such switch for hundreds of ports in reasonable price.
Because of it hadoop tries to run tasks at least in the same rack, if can not do it on the node where data is stored.