Python logging (logutils) with QueueHandler and QueueListener

Question

Logging is needed for a multiprocess python app. Using queues appears to be the best solution. The logutils library provides this.

Is it possible to set the log levels for two handler independently? For example, in the test below I would like STREAM 1 to have WARNING messages and STREAM 2 to have INFO messages. In the test log at the end of the code an INFO message is created that should not output to the console from STREAM 1 handler (WARNING). However, it outputs to both handlers.

For reference, I have been using this page http://plumberjack.blogspot.co.uk/2010/09/improved-queuehandler-queuelistener.html by Vinay Sajip who is the author of the library.

# System imports
import logging
import logging.handlers
try:
    import Queue as queue
except ImportError:
    import queue

# Custom imports
from logutils.queue import QueueHandler, QueueListener

# Get queue
q = queue.Queue(-1)

# Setup stream handler 1 to output WARNING to console
h1 = logging.StreamHandler()
f1 = logging.Formatter('STREAM 1 WARNING: %(threadName)s: %(message)s')
h1.setFormatter(f1)
h1.setLevel(logging.WARNING) # NOT WORKING. This should log >= WARNING

# Setup stream handler 2 to output INFO to console
h2 = logging.StreamHandler()
f2 = logging.Formatter('STREAM 2 INFO: %(threadName)s: %(message)s')
h2.setFormatter(f2)
h2.setLevel(logging.INFO) # NOT WORKING. This should log >= WARNING

# Start queue listener using the stream handler above
ql = QueueListener(q, h1, h2)
ql.start()

# Create log and set handler to queue handle
root = logging.getLogger()
root.setLevel(logging.DEBUG) # Log level = DEBUG
qh = QueueHandler(q)
root.addHandler(qh)

root.info('Look out!') # Create INFO message

ql.stop()

Answer 1

You can use the respect_handler_level argument that's been added to the QueueListener initializer from Python version 3.5, so that the listener will respect handlers individual level.

From QueueListener's doc:

If respect_handler_level is True, a handler’s level is respected (compared with the level for the message) when deciding whether to pass messages to that handler; otherwise, the behaviour is as in previous Python versions - to always pass each message to each handler.

In your case you should replace the initialization of the QueueListener to:

ql = QueueListener(q, h1, h2, respect_handler_level=True)

Answer 2

This is a limitation in the implementation of the QueueListener.handle() method. This is currently:

def handle(self, record):
    record = self.prepare(record)
    for handler in self.handlers:
        handler.handle(record)

To do what you want, it should be

def handle(self, record):
    record = self.prepare(record)
    for handler in self.handlers:
        # CHANGED HERE TO ADD A CONDITION TO CHECK THE HANDLER LEVEL
        if record.levelno >= handler.level:
            handler.handle(record)

I will fix this at some point because I think this is better, but for now you can subclass QueueListener and override the handle method in the subclass.