how to write java Log file using the logger api while using hadoop

Question

i Wrote a map reduce code that i want to debug.

in order to do so i cant use standard output, because the Hadoop platform doesn't print it to the screen unless an error occurs.

instead i tried to use logger, in order to create log file.

i split it to two files using a handler, unfortunately the "severe" log file comes out empty and the general log file only logs things that happens in the main thread and not in the map reduce functions.

the question is as follows:

is there an issue with hadoop and log files or is it a problem with my configuration of the logger? if so how to correct it.

the log configuration code: i use the one logger for the entire application (this time root logger)

public static Logger configureLogging() 
    {
        try
        {
            logger=Logger.getLogger("");
            //FileSystem hdfs=FileSystem.get(URI.create(Misc.S3FS),getConfiguration());
            logger.setLevel(Level.ALL);

            //StreamHandler handler=new StreamHandler(hdfs.create(new Path(Misc.LOGS_PATH+"mylog.log")),new SimpleFormatter());
            FileHandler handler=new FileHandler(Misc.LOGS_PATH+"mylog.xml",true);   
            FileHandler severeHandler=new FileHandler(Misc.LOGS_PATH+"mylogSevere.xml",true);
            severeHandler.setLevel(Level.INFO);
            logger.addHandler(handler);
            logger.addHandler(severeHandler);

        }
        catch (Exception e) 
        {
            e.printStackTrace();
        }
        return logger;

    }

Answer 1

Hadoop comes with preconfigured log4j. All you have to do is import two classes:

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

And now you can define logger inside your mappers, reducers and wherever you want:

private static final Log LOG = LogFactory.getLog(MyClass.class);

And log what you need:

LOG.info("My message");

The messages will show up during job execution. You can tweak log4j configuration with

conf/log4j.properties