Menu
Trying to run the PI example from the hadoop-mapreduce-examples-2.2.0.jar, I'm getting the following exception:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException): Invalid resource request, requested memory < 0, or requested memory > max configured, requestedMemory=1536, maxMemory=512
Not sure where the 1536 came from, but the 512 is the max heap size I set to the child task in mapred-site.xml:
<property>
<name>mapreduce.map.memory.mb</name>
<value>512</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx410m</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>512</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx410m</value>
</property>
What is the correct way to determine the size of map/reduce tasks?
439
YARN uses the MB of memory and virtual cores per node to allocate and track resource usage. For example, a 5 node cluster with 12 GB of memory allocated per node for YARN has a total memory capacity of 60GB. For a default 2GB container size, YARN has room to allocate 30 containers of 2GB each.
You can get to it in two ways: http:/hostname:8088, where hostname is the host name of the server where Resource Manager service runs. Otherwise, from Ambari UI click on YARN (left bar) then click on Quick Links at top middle, then select Resource Manager. You will see the memory and CPU used for each container.
yarn.nodemanager.vmem-pmem-ratioDefines a ratio of allowed virtual memory compared to physical memory. This ratio simply defines how much virtual memory a process can use but the actual tracked size is always calculated from a physical memory limit.
MapReduce requests three different kinds of containers from YARN: the application master container, map containers, and reduce containers. For each container type, there is a corresponding set of properties that can be used to set the resources requested.
512 is the default value of yarn.scheduler.maximum-allocation-mb in yarn-site.xml, and 1536 is the default value of yarn.app.mapreduce.am.resource.mb parameter in mapred-site.xml.
Make sure that allocation-mb > app.mapreduce.am.resource.mb, and it will be ok.
77
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
