Python Scheduled Log Rotating

Question

EDIT:

Looks like other people are having a similar issue with TimedRotatingFileHandler.

Python TimedRotatingFileHandler not rotating at midnight till something new is written to the log file. How to fix this?

Why doesn't my TimedRotatingFileHandler rotate at midnight?

Apparently, the logs don't rotate unless there is some activity happening in the logs. Is there a way I can achieve the scheduled rotating functionality I want using the builtin Handlers without having to create a custom rotator?

---

ORIGINAL POST:

I'm using the TimedRotatingFileHandler of Python's logging module to rotate my logs regularly.

I've specified the configurations in logging.conf for my app's loggers and handlers as such:

[logger_worker]
level=DEBUG
propogate=0
qualname=worker
handlers=workerHandler

[handler_workerHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=standardFormatter
args=("/var/log/app/worker.log","M",1,30,None,False,False)

• Note: for testing purposes I've configured the handler to rotate the logs on every minute, but ideally the logs will be rotated on a daily basis at midnight.

In my app, I am creating the logger like this:

logging.config.fileConfig("logging.conf")
log = logging.getLogger("worker")

log.debug('Hello world')

It is not working as I expected it to work:

1. It is not rotating all the logs
2. It is not rotating every minute as it is configured to do

Observe the ls -l output of the log directory:

-rw-r--r-- 1 root root       0 Apr 19 18:22 dpv.log
-rw-r----- 1 root root    5092 Apr 20 11:47 emperor.log
-rw-r--r-- 1 root root   88939 Apr 20 11:47 uwsgi.log
-rw-r--r-- 1 root root     494 Apr 20 11:46 worker.log
-rw-r--r-- 1 root root   45906 Apr 20 02:08 worker.log.2016-04-20_02-08
-rw-r--r-- 1 root root     494 Apr 20 11:34 worker.log.2016-04-20_11-34
-rw-r--r-- 1 root root     494 Apr 20 11:36 worker.log.2016-04-20_11-36
-rw-r--r-- 1 root root     494 Apr 20 11:44 worker.log.2016-04-20_11-44

What am I doing wrong? Is it possible to rotate the logs on a scheduled term even when nothing is being written to the logs?

Answer 1

Seems TimedRotatingFileHandler doesn't have the functionality that you need. You can try the following function for rotating the log files at the certain time.

def log_rollover(folder_path):
    # folder_path: the absolute path to your log folder 
    today = str(datetime.date.today()).replace('-', '')
    list_of_files = os.listdir(folder_path)
    for log_file in list_of_files:
        file_path = folder_path + '/' + log_file
        if log_file.endswith('.log') and os.stat(file_path).st_size > 0:
            new_file_path = file_path.split('.')[-2] + '-' + today + '.log'
            subprocess.check_call(['cp', file_path, new_file_path])
            subprocess.check_call(['gzip', new_file_path])
            subprocess.check_call(['cp', '/dev/null', file_path]) 

Also you need to use a cron job to run the function at the certain time. APScheduler can be a good option. For example if you want to run the cron job at 3 AM, you can use:

folderPath = '/var/log/app'  # The path to your log folder
scheduler = BlockingScheduler()
scheduler.add_job(log_rollover, trigger='cron', args=[folderPath], hour=3, minute=0)
scheduler.start()

Don't forget to set your timezone for APScheduler.