PermissionError when using python 3.3.4 and RotatingFileHandler

Question

I am trying to get a rotating log file for a GUI application I am writing with python 3.3.4 and PyQt4.

I have the following snippet of code in my main script:

import logging
import resources

logger = logging.getLogger('main.test')

def main():
    logger.setLevel(logging.DEBUG)

    fh = RotatingFileHandler(resources.LOG_FILE_PATH, maxBytes=500, backupCount=5)
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    fh.setFormatter(formatter)

    logger.addHandler(fh)
    logger.info('main')

I have the maxBytes low so that I can test the rotating is working correctly, which it is not. I am getting the following error whenever the log should be rotated:

Traceback (most recent call last):
File "C:\Python33\lib\logging\handlers.py", line 73, in emit
self.doRollover()
File "C:\Python33\lib\logging\handlers.py", line 176, in doRollover
self.rotate(self.baseFilename, dfn)
File "C:\Python33\lib\logging\handlers.py", line 116, in rotate
os.rename(source, dest)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\myuser\\.logtest\\test.log.1'

And nothing is logged. Any help is much appreciated. Thank you

Answer 1

Spent half a day on this as non previous answer resolved my issue.

My working solution is to use https://pypi.org/project/concurrent-log-handler/ instead of RotatingFileHandler. In multiple thread scenarios like Flask app, PermissionError will be raised when we rotate the log file that reaches maximum size.

Install pypiwin32 to get rid of No Module name win32con error.

Thanks go to https://www.programmersought.com/article/43941158027/

Answer 2

Instead of adding handler to the logger object you can directly specify handler in basicConfig(). If you add RotatingFileHandler to the logger object then one object might open the log file and another at the same time might try to rename it which throws the PermissionError.

Below code seems to work pretty well.

import logging
import resources
from logging.handlers import RotatingFileHandler

logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', handlers=[RotatingFileHandler(filename=resources.LOG_FILE_PATH, maxBytes=500, backupCount=5)])
logger = logging.getLogger('main.test')

def main():
    logger.setLevel(logging.DEBUG)
    logger.info('main')