How to parse Yarn logs to obtain performance indicators?

Question

Summary: I need something to yarn logs -applicationId myID | parse.

I am developing a code with some level of parallelism, so I need a feedback about vcores and RAM memory used... I have the application ID of each test, so, after run I can use

 yarn logs -applicationId application_1581298836342_95477 > myYarnLog.txt

but it is a big and complex log, and I need only to check vcores and memory used.

There are a parse for myYarnLog.txt, that filter or calculate the performance indicators?

---

PS: "RAM memory" can be "Aggregate Resource Allocation", vcores can be some virtual-CPU allocation statistics, etc.

Answer 1

in you yarn site.xml similar to, like here

 yarn.resourcemanager.scheduler.monitor.enable

you need silimar in file capacity-scheduler.xml as response here

<property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>
</property>