How to tell Django not to buffer the HTTP POST data?

Question

The client is posting mjpeg stream as HTTP POST:

POST /feed/testfeed HTTP/1.0
Content-type: multipart/x-mixed-replace; boundary=--myboundary

--myboundary
Content-length: 14179
Content-type: image/jpeg

....JFIF....
....

I see no incoming data in Django at all. request.read(6) returns empty string. I add fake "content-Length" header:

POST /feed/testfeed HTTP/1.0
Content-Length: -1
Content-type: multipart/x-mixed-replace; boundary=--myboundary

...

Now it reads the whole data with maximum speed. request.read(6) returns (with the whole data, not just expected 6 bytes) only after I interrupt the connection.

The same behaviour is when I use "PUT" request instead of "POST" one.

How to turn off buffering of the POST request?

Answer 1

It's a little bit of a guess here (because you didn't explain how you are serving the website exactly), but I would think that it's not Django that does the buffering but a web server in front of it. Whether this can be disabled or mitigated depends on the server you're actually using.

You may be interested with the following (in case of Nginx):

• http://forum.nginx.org/read.php?2,94176,94176#msg-94176
• https://serverfault.com/questions/420821/nginx-buffering-data-before-sending-to-fastcgi
• http://nginx.2469901.n2.nabble.com/nginx-proxying-do-not-locally-save-buffer-uploaded-files-td6332400.html

Basically it seems, that with Nginx disabling this behavior might not be possible (see disable request buffering in nginx). Not sure about other servers, but there's plenty of discussion out there, so using google should yield lots of information.