python3: timedrotatingfilehandler log rotation issue with same log file with multiple scripts

Question

Please help in troubleshooting python3 logging from multiple processes into same log file. i have dameon main script, which runs in back ground, and calls few other pythons scripts every 15seconds, and each python script is written with same TimedRotatingFileHandler attributes, and all logs are written into same log file.

everything works fine, except log rotation.

lib/
├── __init__.py
├── FIRST.py
└── SECOND.py
└── THIRD.py
main_daemon.py

main python daemon file looks like

t1 = t2 = t3 = Thread()
my_thread = MYthreads(t1, t2, t3)
################# Daemon Class #################
class Daemon(Daemon):
        def run(self):
             while True:
                my_thread.start_my_class()
                time.sleep(15)

ProcessManager = Daemon('daemon.pid')
ProcessManager.start()

lib/__init__.py file looks lile

class MYthreads:
    def __init__(self, t1, t2, t3):
        self.t1 = t1
        self.t2 = t2
        self.t3 = t3

    def start_my_class(self):
            for file in List_files(path1):
                self.t1 = Thread(target=FIRSTprocess, args=(file,))
                self.t1.start()
                self.t1.join()
            for file in List_files(path2):
                self.t2 = Thread(target=SECONDprocess, args=(file,))
                self.t2.start()
                self.t2.join()
            for file in List_files(path3):
                self.t3 = Thread(target=THIRDprocess, args=(file,))
                self.t3.start()
                self.t3.join()

here the targets are functions imported from seperate python files in lib/ directory. each python files writes logs into same log file.

logging defined in each python file inside lib/ directory..

logger = logging.getLogger('mylogger')
#TimedRoatingFile handler definition
Timed_formatter = logging.Formatter('%(message)s')
Timed_handler = logging.handlers.TimedRotatingFileHandler('my_log', 'midnight', 1, 90)
Timed_handler.setFormatter(Timed_formatter)
logger.addHandler(Timed_handler)
#syslog handler definition
if syslog_streaming and curr_os != 'Windows':
    log_formatter = logging.Formatter('%(name)s: %(message)s')
    Sys_handler = logging.handlers.SysLogHandler(address = '/dev/log')
    Sys_handler.setFormatter(log_formatter)
    logger.addHandler(Sys_handler)
    logger.handlers = [logger.handlers[0], logger.handlers[1]]
else:
    logger.handlers = [logger.handlers[0], ]
# Set lev to DEBUG, applied to both
logger.setLevel(logging.DEBUG)

So, i have defined same logging attributes in all python files, and writing them into same log file, and imported all those functions into __init__.py and calling class every 15seconds through while loop.

Logging & log writing is working fine.. and even syslogging is also working fine.

Problem: Log rotation is not working fine. i must use time rotating handler, since my requirement is to do rollover at 12AM every night. When i check log file after 12 AM, backup file is created, but logs written whole day before 00hrs is lost, i could only few lines of logs generated at 00hours at new date, and older date logs are all LOST.

-rw-r--r--. 1 usertony usertony    1097 Sep  7 01:15 my_log.2020-09-06
-rw-r--r--. 1 usertony usertony    1097 Sep  8 00:00 my_log.2020-09-07
-rw-r--r--. 1 usertony usertony      96 Sep  9 00:00 my_log.2020-09-08

Answer 1

Problem

Your processes are competing to write to the same file, resulting in a race condition, causing issues during rollover.

Solution

You need to define your logger outside of those threads:

https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes

Note

I would not recommend overwriting the Daemon class with your class. If inheritance was the objective name your class something contextual (ie: FileLoggingDeamon).

References

Logging to a single file from multiple processes: https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes

https://stackoverflow.com/a/9993857/806876