Logger hierarchy and the root logger when logging with multiple modules

Question

I have this setup:

main.py
/module
/module/__init__.py (empty)
/module.py

And here is the code for my two files, main.py and module.py respectively:

main.py

import logging
from module import module

logger = logging.getLogger(__name__)

def test():
    logger.warning('in main.py/test')

def main():
    handler = logging.StreamHandler()
    handler.setLevel(logging.INFO)
    formatter = logging.Formatter('%(asctime)s %(name)s/%(module)s [%(levelname)s]: %(message)s', '%Y-%m-%d %H:%M:%S')
    handler.setFormatter(formatter)
    logger.addHandler(handler)

    logger.warning('in main.py/main')
    module.something()

if __name__ == "__main__":
    main()    

module.py

import logging
logger = logging.getLogger(__name__)

def something():
    logger.warning('in module.py/something')

So, what I noticed is that this outputs the following (notice how the module logger has no formatting):

2019-10-01 09:03:40 __main__/main [WARNING]: in main.py/main
in module.py/something

It only seems like only after I make an edit in main.py to change logger = logging.getLogger( __ name __ ) to logger = logging.getLogger() or add logger = logging.getLogger() after def main(): that it logs like this (which is what I want):

2019-10-01 09:04:13 root/main [WARNING]: in main.py/main
2019-10-01 09:04:13 module.module/module [WARNING]: in module.py/something

Why is that? I thought that because main.py is importing module.py, it is naturally higher on the hierarchical scale so module.py would inherit the logger settings as defined in main.py. Do need to explicitly set the root logger (with logger = logging.getLogger()) in main for the inheritance to work? Did I not configure my folder structure correctly to make module.py's logger inherit main.py's logger settings, or is folder structure irrelevant?

The reason I ask is because I thought one should use logger = logging.getLogger( __ name __ ) throughout (even in main.py) and then based on the import structure (or folder structure?), that would determine the hierarchy and loggers would inherit accordingly. And the reason I was making that assumption is because what if I was importing main.py into another program? I guess my point is, I want to make logging as generic as possible such that I can import one module into another and it always inherits the parent's logger settings. Is there a way to display the underlying hierarchy of all the modules for debugging/learning purposes?

Answer 1

The logging hierarchy has nothing to do with file structure in your program. The hierarchy is determined only by the names of the loggers. When you configure a logger, all loggers with its name in the prefix of their name are its children and inherit its configuration unless explicitly stated otherwise.

In your example, logging setup has more to do with execution sequence and the names you've chosen than anything else. When your program runs, it does the following:

1. Runs logging.py from the standard library because of import logging
2. Runs module.py to fulfill from module import module
3. Sets the logger attribute in main to a Logger named __main__.
4. Create a test function
5. Create a main function
6. Run the main function

Some consequences of this sequence of events:

• module.logger is created before main.logger. This doesn't affect the behavior you're seeing, but it's worth noting under the circumstances.
• main.logger is named __main__ if you invoke main as a script. The behavior you see wouldn't change if it was called main, e.g. from python -m main.
• module is clearly not in the same hierarchy as main. Both are descendants of the root logger along different branches.

The last item is really the answer to your question. If you want all the loggers in your program to share the same default logging method, you should configure the root logger, or ensure that they have the same name prefix, which you then configure as if it was the root logger.

You could make all the loggers inherit from main. In module/module.py, you would do

logger = logging.getLogger('__main__.' + __name__)

The issue here is that the name __main__ is hard coded. You don't have a guarantee that it will be __main__ vs main. You could try import main in module so you could do main.__name__ + '.' + __name__, but that wouldn't work as expected. If main was run as __main__, importing it will actually create a second module object with an entirely separate logging hierarchy.

This is why the root logger has no name. It provides exactly the maintainability and consistency you want. You don't have to jump through hoops trying to figure out the root name.

That being said, you should still have main.py logging to the __main__ or main logger. The root logger should only be set up in the import guard. That way, if main is imported as a regular module, it will respect the logging setup of the driver it is running under.

TL;DR

It is conventional to set up the anonymous root logger in the driver of your program. Don't try to inherit loggers from __main__ or the driver module name.