Partial aggregation vs Combiners which one faster?

Question

There are notice about what how cascading/scalding optimized map-side evaluation They use so called Partial Aggregation. Is it actually better approach then Combiners? Are there any performance comparison on some common hadoop tasks(word count for example)? If so wether hadoop will support this in future?

Answer 1

In practice, there are more benefits from partial aggregation than from use of combiners.

The cases where combiners are useful are limited. Also, combiners optimize the amount of throughput required by the tasks, not the number of reduces -- that's a subtle distinction which adds up to significant performance deltas.

There is a much broader range of use cases for partial aggregation in large distributed workflows. Also, partial aggregation can be used to optimize the number of job steps required for a workflow.

Examples are shown in https://github.com/Cascading/Impatient/wiki/Part-5 which uses CountBy and SumBy partial aggregates. If you look back in the code commit history on GitHub for that project, there was previously use of GroupBy and Count, which resulted in more reduces.