mod_wsgi process getting killed and django stops working

Question

I have mod_wsgi running in daemon mode on a custom Linux build. I haven't included any number for processes or threads in the apache config. Here is my config:

WSGIDaemonProcess django user=admin
WSGIProcessGroup django
WSGIScriptAlias /django_apps /django/apache/django.wsgi
WSGIApplicationGroup %{GLOBAL}

On the system, there is a httpd cleanup process running which kills any httpd process that exceeds a certain memory threshold.

It looks like the httpd process which is running mod_wsgi is getting killed. Afterwards, the django portion of my website stops working.

I get this error message:

Script timed out before returning headers: django.wsgi

Every time I access a django page, I get these log messages:

<6> Jul  7 10:13:11 httpd[12598]: [info] mod_wsgi (pid=12598): Initializing Python.
<6> Jul  7 10:13:11 httpd[12598]: [info] mod_wsgi (pid=12598): Attach interpreter ''.
<6> Jul  7 10:13:16 httpd[12638]: [info] mod_wsgi (pid=12638): Attach interpreter ''.
<6> Jul  7 10:13:17 httpd[12615]: [info] mod_wsgi (pid=12615): Destroying interpreters.
<6> Jul  7 10:13:17 httpd[12615]: [info] mod_wsgi (pid=12615): Cleanup interpreter ''.
<6> Jul  7 10:13:17 httpd[12615]: [info] mod_wsgi (pid=12615): Terminating Python.
<6> Jul  7 10:13:17 httpd[12615]: [info] mod_wsgi (pid=12615): Python has shutdown.

Can anybody help me understand what is going here. Why does the mod_wsgi process fail to restart? How do I configure it to restart gracefully?

Thanks very much in advance.

Answer 1

For a start, read:

http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html

and disable initialisation of Python interpreter within embedded processes if only using daemon mode. Those log messages are likely from embedded processes and showing up because of overly aggressive settings for Apache MPM that cause main Apache child processes to be started up and shutdown to much. Disable the unused interpreters and likely those messages will go away.

This issue though is separate to your problem with the daemon mode process.

For that, set on WSGIDaemonProcess the 'display-name=%{GROUP}' option. This will cause daemon mode process to show as '(wsgi:django)' in 'ps' output listing. That way you can clearly identify which process it is.

Then when your process killer kicks in use 'ps' to verify the state of the process and whether there is actually one and whether it shutdown properly. What may be happening is that process is refusing to shutdown because it is hanging on waiting for some child process to shutdown or stuck on some remote file system operation. Alternatively, the code is registering a signal handler which is interfering with correct shutdown.

If that process doesn't shutdown properly it will not be replaced, yet the initiation of a shutdown may put it in to a state where by it doesn't accept new requests either.

So, start with that and report back what you find.

BTW, there is an official mod_wsgi mailing list. StackOverflow is a poor medium for back and forth debugging of problems like this.