Is there a simple Python map-reduce framework that uses the regular filesystem?

Question

I have a few problems which may apply well to the Map-Reduce model. I'd like to experiment with implementing them, but at this stage I don't want to go to the trouble of installing a heavyweight system like Hadoop or Disco.

Is there a lightweight Python framework for map-reduce which uses the regular filesystem for input, temporary files, and output?

Answer 1

http://pythonhosted.org/mrjob/ is great to quickly get started on your local machine, basically all you need is a simple:

pip install mrjob

Answer 2

A Coursera course dedicated to big data suggests using these lightweight python Map-Reduce frameworks:

• http://code.google.com/p/octopy/
• https://github.com/michaelfairley/mincemeatpy

To get you started very quickly, try this example:

https://github.com/michaelfairley/mincemeatpy/zipball/v0.1.2

(hint: for [server address] in this example use localhost)