Hadoop MapReduce log4j - log messages to a custom file in userlogs/job_ dir?

Question

Its not clear to me as how one should configure Hadoop MapReduce log4j at a job level. Can someone help me answer these questions.

1) How to add support log4j logging from a client machine. i.e I want to use log4j property file at the client machine, and hence don't want to disturb the Hadoop log4j setup in the cluster. I would think having the property file in the project/jar should suffice, and hadoop's distributed cache should do the rest transferring the map-reduce jar.

2) How to log messages to a custom file in $HADOOP_HOME/logs/userlogs/job_/ dir.

3) Will map reduce task use both the log4j property file? the one supplied by the client job and the one present in the hadoop cluster? If yes, then the log4j.rootLogger would add both the property values?

Thanks Srivatsan Nallazhagappan

Answer 1

You can configure log4j directly in your code. For example you can call PropertyConfigurator.configure(properties); e.g. in mapper/reducer setup method.

This is example with properties stored on hdfs:

        InputStream is = fs.open(log4jPropertiesPath);
        Properties properties = new Properties();
        properties.load(is);
        PropertyConfigurator.configure(properties);

where fs is FileSystem object and log4jPropertiesPath is path on hdfs.

With this you can also output logs to a dir with job_id. For example you can modify our properities before calling PropertyConfigurator.configure(properties);

Enumeration propertiesNames = properties.propertyNames();
            while (propertiesNames.hasMoreElements()) {
                String propertyKey = (String) propertiesNames.nextElement();
                String propertyValue = properties.getProperty(propertyKey);

                if (propertyValue.indexOf(JOB_ID_PATTERN) != -1) {
                    properties.setProperty(propertyKey, propertyValue.replace(JOB_ID_PATTERN, context.getJobID().toString()));
                }
            }

Answer 2

1. There is no straight forward way to override the log4j properties at each job level.

2. Map Reduce job itself doesn't store the logs in Hadoop,it writes logs in local file system(${hadoop.log.dir}/userlogs) of the datanodes. There is a separate process from Yarn called log-aggregation which collect those logs and combines.

Use yarn logs --applicationId <appId> to fetch the full log, then use unix command to parse and extract the part of the log you need.