Is MapReduce well suited for solving problems in a single-machine multiple-core in-memory environment?

Question

Does the MapReduce abstraction a good one for dealing with problems even in a single machine? For example, I have a 12-core machine and I have to count words in thousands of files (classic MapReduce example).

Using a MapReduce implementation with Mappers and Reducers in multiple threads is a good way to solve this problem, considering that we're working on a single machine with a single Hard-drive?

I guess my question comes down to this: Is the MapReduce paradigm good only for working in a cluster of machines?

Answer 1

In general you can have two situations:

1. Your problem is small enough to fit into the memory of your single system and your single system has enough CPU power to solve the problem within the required time.
2. Your problem is too big. 2.1 Running time is too big (disk IO and/or CPU time) 2.2 Too big to fit into memory (RAM).

For 2.1 and 2.2 the MapReduce paradigm helps a lot in splitting the work into many smaller chunks. If you need more CPU you simply add CPUs.

So if you have a single system and it turns out your problem is too big to fit into memory (point 2.2) you can still benefit from the fact that MapReduce can easily put a part of the problem on disk until that part is to be processed.

The important fact is that if you have a problem that is small enough to fit into memory and small enough to be processed on a single system then a dedicated (non-MapReduce) solution can be a lot faster.