Django/apache handle incomplete/cancelled http requests

Question

When django is deployed on apache with mod_wsgi. It seems to handle incomplete or cancelled requests in a very odd way.

If the client cancels the request, the cancelled request is not cancelled on django, for example, if you are uploading a big file, obviously the body of the request will be actually streamed, so while django is reading the body, and the client cancels the request, it is still processed (just incomplete) and the actual request cancel action is never noticed.

This is one log example from apache, when a request is cancelled. [Fri May 01 22:05:51.055968 2015] [:error] [pid 31609] (70008)Partial results are valid but processing is incomplete: [client 172.31.43.91:3645] mod_wsgi (pid=31609): Unable to get bucket brigade for request.

Then on django code, the actual POST dictionary is never built (because the request is incomplete, yet it arrives to django and is processed as if it had data), and so django will then fail when trying to get data (and return missing XX field errors, or what ever the logic does to handle them)

Finally, when django tries to write back the response, it will obviously fail as well as the client already closed the connection.

This scenario, happens very often on a request that is used as a REST service endpoint for a mobile app. The mobile app uploads large files and so the request is cancelled on app suspend/close, yet the server always seems to get the partial request.

The complete log when this happens would look something like this:

[Fri May 01 22:05:51.055968 2015] [:error] [pid 31609] (70008)Partial results are valid but processing is incomplete: [client 172.31.43.91:3645] mod_wsgi (pid=31609): Unable to get bucket brigade for request.

[Fri May 01 22:05:51.062690 2015] [:error] [pid 10580] some error message related to missing data here

[Fri May 01 22:05:51.068790 2015] [:error] [pid 10580] [remote 172.31.43.91:0] mod_wsgi (pid=10580): Exception occurred processing WSGI script 'some-path/wsgi.py'.

[Fri May 01 22:05:51.068827 2015] [:error] [pid 10580] [remote 172.31.43.91:0] IOError: failed to write data

Now the final question is, is there a way to detect this kind of incomplete request and handle it accordingly, rather than just failing later with missing required data?

Answer 1

When reading request content, if the required length of content had not been read, then the WSGI layer would when wsgi.input.read() was called raise an IOError exception. This may be passed on as is in Django, or in more recent versions be changed to a different derived IOError exception type called UnreadablePostError.

When your application code isn't specifically checking for broken request content and handling that exception type, then it propagates back up and is dealt with by Django as an unhanded exception. Django will at that point attempt to write a 500 error response. When that is written by the WSGI layer, it will fail as the connection had been closed, something that can only be detected by actually attempting to write the response.

So Django should not be giving you incomplete POST data and there should be an exception being raised instead.

As to whether there is a better way of handling it, the answer is no. With the WSGI specification being based around a blocking model, detection and handling of a dropped connections in a clean way isn't really possible. One would need to switch to an ASYNC web server and framework to be able to better handle it and that means not being able to use WSGI or Django.

FWIW, there has been past discussions on the issue of dropped connections on the mod_wsgi mailing list. You might therefore go to Google Groups and search through the archives for the list using search terms like 'dropped connection' or 'failed connection' or 'closed connection' and see what you can find.