How to set a handler for all loggers within a project?

Question

I want to use a memory logger in my project. It keeps track of the last n logging records. A minimal example main file looks like this:

import sys
import logging
from logging import StreamHandler
from test_module import do_stuff

logger = logging.getLogger(__name__)


class MemoryHandler(StreamHandler):
    def __init__(self, n_logs: int):

        StreamHandler.__init__(self)
        self.n_logs = n_logs
        self.my_records = []

    def emit(self, record):
        self.my_records.append(self.format(record))
        self.my_records = self.my_records[-self.n_logs:]

    def to_string(self):
        return '\n'.join(self.my_records)


if __name__ == '__main__':
    logging.basicConfig(stream=sys.stdout, level=logging.INFO)

    mem_handler = MemoryHandler(n_logs=10)
    logger.addHandler(mem_handler)
    logger.info('hello')
    do_stuff()

    print(mem_handler.to_string())

The test module I am importing do_stuff from looks like this:

import logging

logger = logging.getLogger(__name__)


def do_stuff():
    logger.info('doing stuff')

When I run the main function two log statements appear. The one from main and the one from doing stuff, but the memory logger only receives "hello" and not "doing stuff":

INFO:__main__:hello
INFO:test_module:doing stuff
hello

I assume that this is because mem_handler is not added to the test_module logger. I can fix this by adding the mem_handler explicitely:

logging.getLogger('test_module').addHandler(mem_handler)

But in general I don't want to list all modules and add the mem_handler manually. How can I add the mem_handler to all loggers in my project?

Answer 1

The Python logging system is federated. That means there is a tree like structure similar to the package structure. This structure works by logger name and the levels are separated by dots.

If you use the module's __name__ to get the logger it will be equivalant to the dotted name of the package. for example:

package.subpackage.module

In this federated system a message is send up the loggers structure (unless one of the loggers is explicitly configured with propagate=False).

So, the best way to add a handler is to add it to the root logger on the top of the structure and make sure all loggers below propagate.

You can get the root logger with logging.getLogger() (without any name) and then add handlers or other configuration as you like.