Python: correct way to log to the same file from different modules

Question

I have written a bunch of scripts over time and I am in the process of refatoring the scripts to keep the code DRY. I am currently using something along these lines in the various scripts:

if __name__ == '__main__':
    logger = logging.getLogger('dbinit')
    hdlr = logging.FileHandler('/var/logs/tmp/foo.log')
    formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
    hdlr.setFormatter(formatter)
    logger.addHandler(hdlr) 
    logger.setLevel(logging.WARNING)

Rather than repeating this in every script (i.e. "module"), I would like to have this logger initialisation done one somewhere and accessed by the various scripts (Hmm, maybe wrap in a singleton class?).

If I can't do that (i.e. put the logger initialisation code in one core module), I assume that by using the same log file name in the logging.FileHandler() call, the various scripts will write to the same file.

Is this assumption correct?

Last but not the least, what is the best practise (i.e. Pythonic) way to solve this problem?

Answer 1

create a function in your python module like below:

def createLogHandler(job_name,log_file):
    logger = logging.getLogger(job_name)
    ## create a file handler ##
    handler = logging.FileHandler(log_file)
    ## create a logging format ##
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    return logger

Now call the function in you program like this:

job_name = 'dbinit'
log_file = '/var/logs/tmp/foo.log'
logger = createLogHandler(job_name ,log_file )
logger.info('Logger has been created')

Answer 2

Given that you are using if __name__ == __main__, I am assuming that these scripts would be running as different processes. If that's the case, then you should use a separate configuration file.

This configuration can be in stored in a file in the format as specified in the docs. Then, you can use logging.config.fileConfig to load the file. You can also have the configuration stored in JSON/YAML formats, convert it into a dictionary and load it using logging.config.dictConfig. The latter is the current recommended approach, although I find the former more straight forward. Read this for more information.

The advantages of using a config file approach are many fold:

1. It keeps the configuration settings separated from your code
2. It lets non-programmers make changes to configuration, as they are stored in easy to read formats
3. It also keeps you from repeating yourself as you have already mentioned.