Controlling logging functionality in hadoop

Question

How to control logging functionality in hadoop? Hadoop uses default log4j.properties file for controlling logs. My use case is to control logs generated by my classes.

Hadoop daemons like JobTracker, TaskTracker, NameNode and DataNode daemon processes use log4j.properties file from their respective host node’s hadoop-conf-directory. The rootLogger is set to “INFO,console” which logs all message at level INFO to the console.

I trigger hadoop jobs using Oozie Workflow. I tried passing my custom log4j.properties file to the job by setting -Dlog4j.configuration=path/to/log4j.properties system property, but it is not working. Still, it takes log4j properties from the default one.

I am not supposed to touch default log4j.properties file.

I am using Oozie-v3.1.3-incubating, hadoop-v0.20 and cloudera CDH-v4.0.1.

How can I override the default log4j.properties file ?? or How can I control logs for my classes ??

Answer 1

What specifically are you trying to achieve with your own Log4J file? I ask because the logs are distributed across your cluster, but by logging them to the rootLogger, you should be able to see them via the job tracker (by drilling down on the Job task attempts).

If you want to utilize rolling files then you have a difficult time retrieving those files later (again because they are distributed across your task nodes).

If you want to dynamically set log levels, this should be simple enough:

public static Logger log = Logger.getLogger(MyMapper.class);

@Override
protected void setup(Context context) throws IOException,
        InterruptedException {
    log.setLevel(Level.WARN);
}

If you want to add you own appenders, then you should be able to do this programmatically (see this SO Question), in the setup method as above.