TimedRotatingFileHandler doesn't work fine in Django with multi-instance

Question

I use TimedRotatingFileHandler to logging Django log and rotate every day, but check the log file, strange issue is yesterday log is truncated and logging few today's log, yesterday log is lost!

Django 1.4
uwsgi 1.4.9
Python 2.6

I start 8 django instance with uwsgi. The setting.py is

'handlers': {
    'apilog': {
        'level': 'INFO',
        'class': 'logging.handlers.TimedRotatingFileHandler',
        'filename': os.path.join(APILOG, "apilog.log" ),
        'when': 'midnight',
        'formatter': 'info',
        'interval': 1,
        'backupCount': 0,
    },
 },
 'loggers': {                                                                                                                        
    'apilog': {
        'handlers': ['apilog'],
        'level': 'INFO',
        'propagate': True  
     },
  }

Did I miss something? Why old logging is lost?

Answer 1

You should not be logging to a file-based handler from multiple processes concurrently - that is not supported, as there is no portable OS support for it.

To log to a single destination from multiple processes, you can use one of the following approaches:

• Use something like ConcurrentLogHandler
• Use a SysLogHandler (or NTEventLogHandler on Windows)
• Use a SocketHandler which sends the logs to a separate process for writing to file
• Use a QueueHandler with a multiprocessing.Queue, as outlined here.