Setup logging before importing other modules

Question

What is the standard way for setting up the logging system (formatting, level), before modules are imported?

The reason I want to do this is that during the import phase complex data structures are loaded, and progress is shown with loggers. But since the logging level is not set, I do not even see those log entries (level is by default WARNING)

I could do a hack like this:

import logging

logging.basicConfig(...)

import mymodule

But I do not like running code before importing stuff.

Isn't there an accepted way to "pre-configure" the logging system from outside the application, with env variables or any such alternative? Something like:

LOG_LEVEL=INFO LOG_FORMAT="..." python main.py

I can of course evaluate those env vars myself (and at that point my hack above is good enough), but I would like to reuse existing ideas before implementing my own solution.

Answer 1

There is no way to pre-configure the stdlib logging framework with a different default behavior. It does not look at any environment variables.

Best practice is to avoid configuring logging at import time - it is up to the user to decide the appropriate log level and where the logs should go to (if anywhere). For the logging configuration to be, well, configurable it needs to be delayed until the entry-point of your application rather than eagerly configured at import time.

The library code (mymodule) should not assume that there is a terminal that somebody is watching for output, or even that there is a writable filesystem available for a log file.

I would recommend this pattern:

# main.py
import logging
import mymodule

def main():
    # parse cmdline args here
    logging.basicConfig(...)
    # or logging.config.fileConfig
    # or logging.config.dictConfig
    mymodule.init()
    # do work here

if __name__ == "__main__":
    main()

And in the library code:

# mymodule.py
import logging

log = logging.getLogger(__name__)

def init():
    log.info("loading complex data structures...")

This avoids dropping the initialization log events on the floor, but there is another benefit to an explicit init call in the library code: the dataset for initialization can be configurable, via arguments to init() or a user-provided config file. This means during testing you can load from a smaller scaled-down dataset, so that the test suite does not run slowly.