Python logging - new log file each loop iteration

Question

I would like to generate a new log file on each iteration of a loop in Python using the logging module. I am analysing data in a for loop, where each iteration of the loop contains information on a new object. I would like to generate a log file per object.

I looked at the docs for the logging module and there is capability to change log file on time intervals or when the log file fills up, but I cannot see how to iteratively generate a new log file with a new name. I know ahead of time how many objects are in the loop.

My imagined pseudo code would be:

import logging

for target in targets:
    logfile_name = f"{target}.log"
    logging.basicConfig(format='%(asctime)s - %(levelname)s : %(message)s',
                        datefmt='%Y-%m/%dT%H:%M:%S',
                        filename=logfile_name,
                        level=logging.DEBUG)


    # analyse target infomation
    logging.info('log target info...')

However, the logging information is always appended to the fist log file for target 1.

Is there a way to force a new log file at the beginning of each loop?

Answer 1

Rather than using logging directly, you need to use logger objects. Go thorough the docs here.

Create a new logger object as a first statement in the loop. The below is a working solution.

import logging
import sys


def my_custom_logger(logger_name, level=logging.DEBUG):
    """
    Method to return a custom logger with the given name and level
    """
    logger = logging.getLogger(logger_name)
    logger.setLevel(level)
    format_string = ("%(asctime)s — %(name)s — %(levelname)s — %(funcName)s:"
                    "%(lineno)d — %(message)s")
    log_format = logging.Formatter(format_string)
    # Creating and adding the console handler
    console_handler = logging.StreamHandler(sys.stdout)
    console_handler.setFormatter(log_format)
    logger.addHandler(console_handler)
    # Creating and adding the file handler
    file_handler = logging.FileHandler(logger_name, mode='a')
    file_handler.setFormatter(log_format)
    logger.addHandler(file_handler)
    return logger


if __name__ == "__main__":
    for item in range(10):
        logger = my_custom_logger(f"Logger{item}")
        logger.debug(item)

This writes to a different log file for each iteration.

Answer 2

This might not be the best solution, but it will create new log file for each iteration. What this is doing is, adding a new file handler in each iteration.

import logging
targets = ["a", "b", "c"]
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

for target in targets:
    log_file = "{}.log".format(target)
    log_format = "|%(levelname)s| : [%(filename)s]--[%(funcName)s] : %(message)s"
    formatter = logging.Formatter(log_format)

    # create file handler and set the formatter
    file_handler = logging.FileHandler(log_file)
    file_handler.setFormatter(formatter)

    # add handler to the logger
    logger.addHandler(file_handler)

    # sample message
    logger.info("Log file: {}".format(target))