Different levels of logging in python

Question

I want to do something which I thought is simple.

Actually with the python logging module, I am interested logging everything on the command line at the level given from the command line arguments, and log to file to a fixed DEBUG level.

Creating two different loggers with different levels doesn't work, but setting the levels of two different handlers both added to the root logger doesn't work either, so any idea about how I should actually do it? (reading on other links the second approach should work, so am I doing something else stupid?)

This is the code which sets up my logging system at the moment:

class LoggerSetup(object):
    """Setup the different logger objects
    """

    def __init__(self):
        self.root_logger = logging.getLogger()
        self.shell_hdlr = logging.StreamHandler()

    #TODO: add another logging handler which stores to a temporary file
    #which should be cleaned up later
    def setup_shell_logger(self, log_level):
        self.root_logger.setLevel(LOG_LEVELS[log_level])
        # in this way the root logger is not set but the handlers are set
        self.shell_hdlr = logging.StreamHandler()
        self.shell_hdlr.setLevel(LOG_LEVELS[log_level])
        self.shell_hdlr.setFormatter(StarFormatter())
        #FIXME: add the support for regular expression exclusion too
        self.root_logger.addHandler(self.shell_hdlr)

    def setup_log_include(self, log_include):
        """Set up the filter to include log messages
        """
        if log_include:
            incl = FilterInclude(log_include)
            self.shell_hdlr.addFilter(incl)

    def setup_log_exclude(self, log_exclude):
        """Set up the filters to exclude log messages
        """
        if log_exclude:
            excl = FilterExclude(log_exclude)
            self.shell_hdlr.addFilter(excl)

    def setup_file_logging(self):
        """Set up the file logger, which always logs in DEBUG mode
        even if the top level logger is set to another level
        """
        #XXX: not working, one possible way to make it work is to create
        #only one log, and different handler/filters to make to handle all
        #the different outputs
        file_handler = logging.FileHandler(LOG_FILENAME)
        # the file logging is always in debug mode
        file_handler.setLevel(logging.DEBUG)
        formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s - %(asctime)s')
        file_handler.setFormatter(formatter)
        self.root_logger.addHandler(file_handler)

Answer 1

This is something I'm using with all of my Python command-line apps. It's a little verbose, but you should be able to get a logger that accepts an optional argument to create a console logger at any level, irrespective of what's being logged to the file:

#!/usr/bin/env python
import logging
from argparse import ArgumentParser

COMPANY_LOGGER = 'COMPANY.Python.Logger'
CONSL_LEVEL_RANGE = range(0, 51)
LOG_FILE = 'company.log'
FORMAT_STR = '%(asctime)s %(levelname)s %(message)s'

parser = ArgumentParser()
parser.add_argument('-c', '--console-log', metavar='ARG',
                    type=int, choices=range(0, 51),
                    action='store', dest='console_log',
                    default=None,
                    help='Adds a console logger for the level specified in the range 1..50')

args = parser.parse_args()

# Create logger
logger = logging.getLogger(COMPANY_LOGGER)
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter(FORMAT_STR)

# Add FileHandler and only log WARNING and higher
fh = logging.FileHandler(LOG_FILE)
fh.name = 'File Logger'
fh.level = logging.WARNING
fh.formatter = formatter
logger.addHandler(fh)

# Add optional ConsoleHandler
if args.console_log:
    ch = logging.StreamHandler()
    ch.name = 'Console Logger'
    ch.level = args.console_log
    ch.formatter = formatter
    logger.addHandler(ch)

logger.debug('DEBUG')
logger.info('INFO')
logger.warning('WARNING')
logger.critical('CRITICAL')

Whenrun from the command line we can see the differences in logged levels.

-c1 equates to "DEBUG and higher" (the most verbose), but company.log is still only logging WARNING and higher:

~ zacharyyoung$ ./so.py -c1
2012-01-12 08:59:50,086 DEBUG DEBUG
2012-01-12 08:59:50,086 INFO INFO
2012-01-12 08:59:50,087 WARNING WARNING
2012-01-12 08:59:50,087 CRITICAL CRITICAL

~ zacharyyoung$ cat company.log 
2012-01-12 08:59:50,087 WARNING WARNING
2012-01-12 08:59:50,087 CRITICAL CRITICAL

-c20 equates to INFO:

~ zacharyyoung$ ./so.py -c20
2012-01-12 09:00:09,393 INFO INFO
2012-01-12 09:00:09,393 WARNING WARNING
2012-01-12 09:00:09,393 CRITICAL CRITICAL

~ zacharyyoung$ cat company.log 
2012-01-12 08:59:50,087 WARNING WARNING
2012-01-12 08:59:50,087 CRITICAL CRITICAL
2012-01-12 09:00:09,393 WARNING WARNING
2012-01-12 09:00:09,393 CRITICAL CRITICAL