python logging: Custom Python LogRecord Throwing an error

Question

I am trying to use the python logging module to create a custom log file that records other information like host name and adds it to my DB. Below are the classes I created to do this, and the Handler part was working just fine, but now that I added a custom LogRecord class, it throws this error:

/src/lib/__init__.py", line 31, in __init__
logging.LogRecord.__init__(self, *args, **kwargs)
exceptions.TypeError: __init__() takes at most 9 arguments (10 given)

And here is how I execute it

logging.setLoggerClass(MyLogger)

log = logging.getLogger('testing')
log.addHandler(MyLogHandler())
d = {'host': '192.168.0.1'}
log.warn('Hi', d)    

And here are the classes. It obviously has to do with the *args, **kwargs, but when I look at it, the *args is empty, and **kwargs only contains the d variable specified above. I don't understand the problem.

class MyLogRecord(logging.LogRecord):
    def __init__(self, *args, **kwargs):
        logging.LogRecord.__init__(self, *args, **kwargs) //THIS IS THE LINE IT DIES ON
        self.host = 'localhost'

class MyLogFormatter(logging.Formatter):

    def __init__(self, fmt, datefmt=None, host=None):
        logging.Formatter.__init__(self, fmt, datefmt)
        self.host = host

    def format(self, record):
        return logging.Formatter.format(record)

class MyLogger(logging.getLoggerClass()):
    def makeRecord(self, *args, **kwargs):
        return MyLogRecord(*args, **kwargs)

class MyLogHandler(logging.Handler): # Inherit from logging.Handler
    def __init__(self):
        # run the regular Handler __init__
        logging.Handler.__init__(self)
        # Our custom argument
        self.mongo = MongoLogger()

    def setupCustomLogger(self, name, this_host):
        formatter = MyLogFormatter(fmt='%(asctime)s - %(levelname)s - %(module)s - %(message)s - %(host)s')

        handler = logging.StreamHandler()
        handler.setFormatter(formatter)

        logger = logging.getLogger(name)
        logger.setLevel(logging.DEBUG)
        logger.addHandler(handler)
        return logger

    def emit(self, record):
        # record.message is the log message
        self.mongo.log(record)


class MongoLogger(object):

'''Logs messages to a MongoDB fh_admin log collection.'''
def log(self, message):
    #@todo write log to DB
    print message

Answer 1

The error is telling you exactly what's wrong; you are calling the constructor with too many arguments. To see what I mean, take a look at how log-records are ordinarily constructed in the default implementation of makeRecord:

def makeRecord(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None):
    """
    A factory method which can be overridden in subclasses to create
    specialized LogRecords.
    """
    rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
    if extra is not None:
        for key in extra:
            if (key in ["message", "asctime"]) or (key in rv.__dict__):
                raise KeyError("Attempt to overwrite %r in LogRecord" % key)
            rv.__dict__[key] = extra[key]
    return rv

Notice how makeRecord takes an extra param that it doesn't pass directly to LogRecord? You, on the other hand, are passing that directly to LogRecord.__init__, which is causing the error.

From here, you've got two options; you could provide a more complete implementation of makeRecord, or you could try using the LoggerAdapter class which should help you achieve the same goal with less code.

Here's an example:

# Common log info to be added to all logs reported with `log_adapter`
context = {'host': 'localhost'}

log = logging.getLogger('testing')
log.addHandler(logging.StreamHandler())
d = {'host': '192.168.0.1'}

log_adapter = logging.LoggerAdapter(log, context)
log_adapter.warning('Hi', d)

If you need to calculate the value of 'host' (for example) each time something is logged, you could make context an instance of a class that looks like a dictionary. Like so:

class LogContext(object):

    def __getitem__(self, key):
        if key == 'host':
            return 'localhost'
        raise KeyError(key)

    def __iter__(self):
        return iter(['host'])

log_adapter = logging.LoggerAdapter(log, LogContext())
log_adapter.warning('Hi', d)

One thing to note about LoggingAdapter, it apparently doesn't define all of the handy shortcut functions as the ordinary Logger class. That's why I've called the warning method instead of warn as you did above.

More info on LoggingAdapter and adding context to your logs can be found in the python docs.

NOTE - I didn't include MyLogHandler, MyLogFormatter, or MongoLogger in my examples as they were not relevant to the issue/error.