More than 120 counters in hadoop

Question

There's a limit for Hadoop counter size. It's 120 by default. I try to use the configuration "mapreduce.job.counters.limit" to change that, but it doesn't work. I've seen the source code. It's like the instance of JobConf in class "org.apache.hadoop.mapred.Counters" is private. Have anybody seen that before? What's your solution? THX :)

Answer 1

You can override that property in mapred-site.xml on your JT, TT, client nodes but make sure that this will be a system-wide modification:

<configuration>
  ...
  <property>
    <name>mapreduce.job.counters.limit</name>
    <value>500</value>
  </property>
  ...
</configuration>

Then restart the mapreduce service on your cluster.