How to limit the number of reduce jobs in mapreduce java code in hadoop

Question

I'm new to Hadoop and I want to limit the number of reduce jobs in my application.

In the cluster, the maximum number of reduce jobs is 120. But, I don't want to use all of them, because my application doesn't need that many number of reduce jobs.

I tried the solution below, but it didn't change anything . My application still uses 120 reduce jobs.

How can I set the number of reduce jobs?

https://stackoverflow.com/questions/33237361/unable-to-set-mapreduce-job-reduces-through-generic-option-parser

Thanks for your response.

Answer 1

From job class,

job.setNumReduceTasks(100);

From shell command

hadoop jar yourJar.jar -D mapred.reduce.tasks=1 <input> <output>

Answer 2

Please check below values

With 1.x

mapred.reduce.tasks will take precedence unless you specify job.setNumReduceTasks(x);

With 2.x

mapreduce.job.reduces will take precedence unless you specify job.setNumReduceTasks(x);

You can implement suggestion by Ramzy with -D parameter. Check the hadoop version and respective property before you set the property.

Have a look at deprecated properties and new properties between 1.x and 2.x versions