Python TimedRotatingFileHandler overwrites logs

Question

I setup TimedRotatingFileHandler like that:

import logging as logging
from logging.handlers import TimedRotatingFileHandler
import os
import time

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

# new file every minute
rotation_logging_handler = TimedRotatingFileHandler('logs/log', 
                               when='m', 
                               interval=1, 
                               backupCount=5)
rotation_logging_handler.setLevel(logging.DEBUG)

format = u'%(asctime)s\t%(levelname)s\t%(filename)s:%(lineno)d\t%(message)s'
rotation_logging_handler.setFormatter(logging.Formatter(format))
rotation_logging_handler.suffix = '%Y-%m-%d'

logger.addHandler(rotation_logging_handler)

Usage:

logger.logging.info('Service started at port %s', config.get_param('port'))

while True:
    time.sleep(21)
    logger.logging.info('Now time is {}'.format(time.time()))

I expected that every minute new messages from logs/log had to append to existing log file for current date. Instead it every minute messages from logs/log overwrote existing log file for current date.

What should I do to reach that behaviour?

PS: After small research I found that TimedRotatingFileHandler in the doRollover method deletes existing log file and creates new file. So first solution is to create new handler derived from TimedRotatingFileHandler which creates new file (with some index for example) insted of deleting existing log file.

Answer 1

It is very much possible to change the filenames when they are rotated to anything you want by overrding the rotation_filename method of the BaseRotatingHandler class and setting it to an appropriate callable.

Here is a very trivial example of doing the same, but you can tweak it to suit your needs.

import logging
from logging.handlers import RotatingFileHandler
import datetime as dt

def filer(default_name):
    now = dt.datetime.now()
    fname, ext = default_name.split(".")[:-1]
    return f"{fname}_{now.strftime('%Y-%m-%d-%H:%M:%S')}.{ext}"

logger = logging.getLogger()
rotating_file_handler = RotatingFileHandler(filename="/Users/rbhanot/file.txt",
                                            mode="a",
                                            maxBytes=201,
                                                 backupCount=5)
rotating_file_handler.rotation_filename = filer
formatter = logging.Formatter(
    '%(asctime)s %(name)s:%(levelname)s - %(message)s')
rotating_file_handler.setFormatter(formatter)
logger.addHandler(rotating_file_handler)
logger.setLevel(logging.INFO)

logger.info("hello world")

And this is the result you will get out of it

❯ /bin/ls ~/file*
file.txt  file_2025-02-13-17:16:49.txt. file_2025-02-13-17:17:16.txt