Django with wsgi sporadically fails with "Premature end of script headers:"

Question

I have a site that is running for months now on Apache2 and periodically it will just hiccup with the following:

[Sat Nov 12 06:18:34 2011] [error] [client X.Y.Z.158] Premature end of script headers: sleepsoundly_wsgi.py
[Sat Nov 12 06:18:49 2011] [error] [client X.Y.Z.158] Premature end of script headers: sleepsoundly_wsgi.py

It has run 1000s of requests without a problem, but periodically it will do this a couple of times and then everything will be ok. This is happening while uploading about 300 files (.5 MB each). Each file is upload separately, 3 files at a time, 225 files upload fine, 226 and 227 failed, and then 228 -> end all worked normally. It doesn't do this every time, just occasionally and it isn't always these files that fail. Another time file #291 failed and all the rest worked.

I've got nothing in the log to go on other than this cryptic message.

I've checked and the only version of python on the machine is 2.7.1. I don't get an email from django, I don't get any of the normal clues as to what might be happening. I'm curious how to start troubleshooting this one. It recovers on its own, the automated program uploading the files keeps moving. How do I figure out what is going on in this case?

Server version: Apache/2.2.17 (Ubuntu)
Server built:   Sep  1 2011 09:25:26
mod_wsgi: Version: 3.3-2ubuntu2

Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)

wsgi.conf has no lines in it that are not commented out.

VirtualHost setup:
    WSGIDaemonProcess myemr user=mjones processes=1 maximum-requests=500 threads=15
    WSGIProcessGroup  myemr
    WSGIScriptAlias   / /var/www/Python/myemr/myemr/deploy/myemr_wsgi.py

myemr_wsgi.py
    from os.path import abspath, dirname, join
    import sys

    # For packages that don't play well with mod_wsgi
    sys.stdout = sys.stderr

    sys.path.insert(0, abspath(join(dirname(__file__), "../..")))
    sys.path.insert(0, abspath(join(dirname(__file__), "../../myemr")))
    sys.path.insert(0, abspath(join(dirname(__file__), "../../myemr/apps")))
    sys.path.insert(0, abspath(join(dirname(__file__), "../../lib/python2.7/site-packages/")))

    # We have to add both of these because they are  installed with git?
    sys.path.insert(0, abspath(join(dirname(__file__), "../../src/pinax/")))

    from django.core.handlers.wsgi import WSGIHandler
    import pinax.env

    # setup the environment for Django and Pinax
    pinax.env.setup_environ(project_path='myemr')

    # set application for WSGI processing
    application = WSGIHandler()

Answer 1

The problem is likely because you have 'maximum-requests' set to 500. This will cause the mod_wsgi daemon processes to be restarted periodically. Since the mod_wsgi daemon processes are running in multithreaded configuration, if there was a stuck request, or long running request which doesn't finish before the forced shutdown timeout when the restart is being done, then it will be aborted by virtue of the process restart. The Apache child worker process will then see that as a request terminated without returning headers with the message you are seeing.

Moral of the story, DO NOT use 'maximum-requests' options in production systems unless you have a very good reason such as out of control memory growth.

Version 4.0 of mod_wsgi will have an optional but slightly more graceful restart option which can be applied for this case, but it still can't wait forever and a stuck request will still have to be aborted at some point and you will still see the message.

BTW, don't specify 'processes=1' as it defaults to one process anyway and any use of the 'processes' option, even with value '1', causes 'wsgi.multiprocess' to be set True. Use of 'processes=1' should only be done where have a single process AND you have many Apache instances which you are load balancing across, also with a single process.