What is the maximum value for mapreduce.task.io.sort.mb?

Question

When I set mapreduce.task.io.sort.mb = 100000. I get following exception.

java.lang.Exception: java.io.IOException: Invalid "mapreduce.task.io.sort.mb": 100000

What is the maximum value for mapreduce.task.io.sort.mb?

Answer 1

I realize this question is old, but for those asking the same question you can check out some of the bugs around this value being capped

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.3/bk_releasenotes_hdp_2.1/content/ch_relnotes-hdpch_relnotes-hdp-2.1.1-knownissues-mapreduce.html

BUG-12005: Mapreduce.task.io.sort.mb is capped at 2047.

Problem: mapreduce.task.io.sort.mb is hardcoded to not allow values larger than 2047. If you enter a value larger then this the map tasks will always crash at this line:

https://github.com/apache/hadoop-mapreduce/blob/HDFS-641/src/java/org/apache/hadoop/mapred/MapTask.java?source=cc#L746

Answer 2

hadoop-2.6.0 org.apache.hadoop.mapred.MapTask.java

line 427:we can't set mapreduce.task.io.sort.mb exceed 2047