Connection resets on (large) file upload

Question

So I'm having a rather vexing issue with file uploads in Pylons that I'm hoping someone might be able to assist me with. Originally I posted the issue here thinking this was an issue with Pylons, but I've since come to conclude that the issue is due to the server, i.e., paste, and I've tracked down exactly where the issue occurs, but am unsure how to resolve it. The behavior is as follows. I have a Pylons web application that I'm serving via paster. One piece of the application allows for users to upload files to the systems, and sometimes these can be rather big (> 50mb). During such uploads, the server sporadically resets the connection (i.e., in Chrome the error reads "Error 101 (net::ERR_CONNECTION_RESET): The connection was reset.".) This does not always occur, only about ~50% of the time with larger files.

On these cases, the server never hands the POST up to the application level. No Exception is thrown (at least, that I can find). I've narrowed the issue down to cascade.py (in particular, the wsgi_application method invoked from httpserver.py). From the latter:

result = self.server.wsgi_application(self.wsgi_environ, self.wsgi_start_response)

In the call function of cascade.py, the file appears to be written to a temporary file. The relevant loop begins at:

while copy_len > 0:

It then writes chunks one by one to the open file. None of these writes individually throws an exception. However, the server will often simply 'go away' during these writes, seemingly at random (time does not seem to be correlated with this, either). So if I print out the copy_len size, which is being decremented, I watch it decrease up to a point, and then the server all of a sudden sends a 'connection reset' (or at least the client thinks it does) and the method seems to be exited (the copy_len ceases to be printed, as if the loop has been exited, though it was not at 0).

I'm completely stumped, as 1) this only happens sporadically, and 2) I can't seem to track down an actual error/Exception -- it just sort of... dies.

If it matters, I'm running paste 1.7.3, Python 2.7 and Fedora OS. Does anyone have any insight into what might be happening, and how I might be able to resolve this? Thank you very much for any help.

Answer 1

Two ideas come to mind ...

• Test with another more robust webserver, such as nginx. It can be run concurrently if need be. See if the problem persists. While paste can handle web apps it isn't meant to handle huge file transfers as far as I know.
• Are you using Chunked Transfer Encoding during the upload?