How to log to a single file with concurrent processes in Django without exclusive locks

Question

Given a Django application that is being executed concurrently on multiple servers, how can this application log to a single shared log file (in a network share), without keeping this file permanently open in exclusive mode?

This situation applies to Django applications hosted on Windows Azure Websites when you want to take advantage of log streaming.

On this sample project, I've trying using ConcurrentLogHandler like this:

in settings.py:

'ConcurrentLogHandler':{
    'level': 'DEBUG',
    'class': 'cloghandler.ConcurrentRotatingFileHandler',
    'formatter': 'verbose',
    'filename': os.getenv('LOGFILE', 'django.log')
},

in views.py:

from time import gmtime, strftime
import logging
from django.http import HttpResponse

logger = logging.getLogger(__name__)

def home(request):
    current_time = strftime("%Y-%m-%d %H:%M:%S", gmtime())
    logger.info('home ' + current_time)
    return HttpResponse("Hello from Django! It is now " + current_time + ".\n")

The logs are written but the file doesn't seem to be flushed while the website is running. Also, if I try to read the file using FTP I get this message: "550 The process cannot access the file because it is being used by another process."

If I stop the application, the file is closed and I can read the file and see all the logs in it.

I assume that ConcurrentLogHandler would allow shared access to the log file. Is this assumption wrong? Is there some additional configuration needed? Is there an alternative?

Answer 1

An alternative would be for all Django logging to be sent to a queue (e.g. a Redis queue, using something like this, or a multiprocessing.Queue) and then a single process reads the queue and writes records to file. There are more moving parts, so this may or may not be appropriate for your needs, but it would eliminate the file contention. See this post for more options when using logging from multiple processes.

You can of course also set up a socket server and use a SocketHandler to send logging events from all Django processes to the server, which writes to file. The Python docs contain a working example of such a server.

Answer 2

A scalable pure-logging approach (as in using only system logging components) can be implemented using syslog-ng

syslog-ng is syslog++, so it will not break any syslog configurations and logging if you already have some. Installation on Ubuntu is straightforward:

sudo apt-get install syslog-ng

for installation on windows, refer:http://www.syslog.org/logged/running-syslog-ng-on-windows/

Multiple django applications or multiple servers log to syslog-ng which uses UDP or TCP (depending on your configuration) to send it to a central syslog-ng server which logs it on that machine. You can use a basic regex to identify the application in syslog-ng and route it accordingly.

Django logging configuration

'syslog-verbose': {
    'format': 'myapp %(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
}

'ConcurrentLogHandler':{
    'level': 'DEBUG',
    'class': 'logging.handlers.SysLogHandler',
    'formatter': 'syslog-verbose',
},

Note the "myapp" string added to the formatter to identify the application by syslog-ng.

You can even configure nginx, apache and other servers to log to syslog-ng. E.g. for apache:

CustomLog "| /usr/bin/logger -t 'apache' -u /var/run/apache_access_log.socket" combined

Syslog-ng client configuration

Append this to the end of /etc/syslog-ng/syslog-ng.conf

filter filter_loghostclient {
    program("^myapp$");
};

destination dest_loghostclient {
    tcp("destination_logserver.example.com" port (514));
};

log {
    source(s_all);
    filter(filter_loghostclient);
    destination(dest_loghostclient);
};

source s_apache_access {
    #apache access logs getting written directly to a socket (as described above)
    unix-stream("/var/run/apache_access_log.socket" max-connections(512) keep-alive(yes)); 
};
log{
    source(s_apache_access);
    destination(dest_loghostclient);
};

filter f_apache_err {
    #Apache error logs
    program("apache") and level(err); 
}; 

log{
    source(s_all);
    filter(f_apache_err);
    destination(dest_loghostclient);
};

Syslog-ng aggregator configuration

Append the following to the syslog-ng config file on destination_logserver.example.com

source src_loghostserver {
    tcp(port(514) keep-alive(yes) max_connections(1000));
};

destination dest_loghostserver {
    file("/var/log/myproject/request_\$R_YEAR\$R_MONTH\$R_DAY.log");
};

log {
    source(src_loghostserver);
    destination(dest_loghostserver);
};