Menu
I want to ask, after I've ran my hadoop mapreduce application, how can I get the total memory and CPU usage of that application. I've seen it on log and resource manager web page but I didn't get it.
Is it possible? Can I get it per job execution or on per application, and can I get it per node usage or total usage?
Thanks a lot....
929
YARN uses the MB of memory and virtual cores per node to allocate and track resource usage. For example, a 5 node cluster with 12 GB of memory allocated per node for YARN has a total memory capacity of 60GB. For a default 2GB container size, YARN has room to allocate 30 containers of 2GB each.
YARN also allows different data processing engines like graph processing, interactive processing, stream processing as well as batch processing to run and process data stored in HDFS (Hadoop Distributed File System) thus making the system much more efficient.
A container will become reserved state when the container is assigned to some nodemanager node which do not have enough resource(cpu or memory) for it.
Yes, you can very well check the total memory and cpu usage of the application.
You may use the Job tracker UI, click on the counters link on the completed page, and might get a typical view as follows. The memory and cpu counters were highlighted.
Clicking on the highlighted links will show you the consumption by every Tasks (Map/Reduce) of the job.
To view these values of these consumption counters from command line, you may use
hadoop job -counter <job-id> org.apache.hadoop.mapreduce.TaskCounter CPU_MILLISECONDS for cpu usage
hadoop job -counter <job-id> org.apache.hadoop.mapreduce.TaskCounter PHYSICAL_MEMORY_BYTES for memory usage
144
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
