How to append the logs in python

Question

There are two files named file1 and file2 in the python project.

In file1.py , Iam capturing the log to file 'sample.log' and executing the file2.py through os.system() command. In file2.py, I am opening the sample log file sample.log in appending mode and sending logs to that file.

I have executed the file1.py, even though I have opened the log file(file2.py) in append mode , I am unable to see the file2.py log. File2 contents are overridden by file1 contents.

How Can I append file2.py logs with file1.py logs in sample.log?

file1.py

import logging
import os

logFile = 'sample.log'
logging.basicConfig( filename = logFile,filemode = 'w',level = logging.DEBUG,format = '%(asctime)s - %(levelname)s: %(message)s',\
                     datefmt = '%m/%d/%Y %I:%M:%S %p' )

logging.debug("Starting of the file1")

os.system("python file2.py")

for i in range(0,2):
    logging.debug("Iam in file1")

file2.py:

import logging

logFile = 'sample.log'
logging.basicConfig( filename = logFile,filemode = 'a',level = logging.DEBUG,format = '%(asctime)s - %(levelname)s: %(message)s',\
                     datefmt = '%m/%d/%Y %I:%M:%S %p' )

for i in range(0,2):
   logging.debug("Iam in file2")

Need the output like this

03/06/2015 08:02:03 PM - DEBUG: Starting of the file1

03/06/2015 08:02:03 PM - DEBUG: Iam in file2

03/06/2015 08:02:03 PM - DEBUG: Iam in file2

03/06/2015 08:02:03 PM - DEBUG: Iam in file1

03/06/2015 08:02:03 PM - DEBUG: Iam in file1

but getting like this:

03/06/2015 08:02:49 PM - DEBUG: Starting of the file1

03/06/2015 08:02:49 PM - DEBUG: Iam in file1

03/06/2015 08:02:49 PM - DEBUG: Iam in file1

Could someone help in this to get my expected output, Thanks in Advance

Answer 1

There are two main faults with your approach:

1. There should only be one logger on a file. If you are going to use os.system you'll have to use two log files because you can't open a duplicate Logger handle in file2.py.

2. You have different write-modes which causes a race condition. It could be that your filemode='a' completes all of its writes and then filemode='w' overwrites the log file from the start, clearing all previously written "file2" logging output.

Answer 2

You are getting only logs from file1 (and sometimes few lines from file2 at the end of the logfile) because whenever file2 writes something to the log it (roughly) first seeks to the end of the file and then writes the data (filemode='a') but when file1 writes to the file it only writes the data and its position in the file is not updated and it overwrites any lines from file2 (filemode='w').

What you want is use filemode='a' in both files.