Python TimedRotatingFileHandler - logs are missing

Question

I am running my python application in apache environment and using timedRotatingFileHandler to log. I have setup logger in a way that It is supposed to rotate midnight everyday. My all processes writes into the same logger file. Somehow logger is missing to log info at times. And sometimes I see logger writing into two files (old file and rotated file) at the same time.

I couldn't able to understand why is this happening? Doesn't TimedrotatingFileHandler work in multiprocess enivironment? If not why is that so?

Please help me to understand..

Answer 1

You can't log (and rotate) to the same file from multiple processes naively because OS wouldn't know how to serialize the write and rotate instructions from 2 different processes. What you are experiencing is known as a race condition as 2 processes are competing to write to the same file and close it and open with a new file handle at the same time at rotation time. Only 1 process will win a new file handle when you rotate, so that may explain the missing log event.

Here's a recipe from Python's documentation with hints about how to log to the same place.

http://docs.python.org/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes

Essentially you will want to have a separate process listening to logging events coming from multiple places and then that process will log the events to a single file. You can configure rotation in that listener process too.

If you are not sure how to write this, you can try using a package such as Sentry or Facebook's Scribe. I recommend Sentry because Scribe is not trivial to setup.