S3 Upload with pycurl interrupts

Question

I'm using pycurl as a back-end for the boto Python library. It is very fast and versatile but I have the problem that uploads of big files very often fail on connection reset. When I use plain boto with plain httplib, it is much more reliable.

What I found using Wireshark is that after a while (or sometimes rather soon as well), my machine stops receiving ACKs from S3, so it resets the connection. It almost seems that pycurl is so fast that it chokes the connection. And if I throttle the upload (I use the multi interface) or use a slower internet connection, the upload runs fine.

I'm still wondering what could I have possibly done wrong.

I also tried to upload with the .NET S3 SDK. It's about 3 times slower, but succeeds. Also, it's all on Windows 7, OS X machine on the same network again uploads a lot slower, but reliably.

Answer 1

Since you mentioned that you experience the problem on Windows 7, can you run a command prompt as administrator and post the results of netsh int tcp show global? You should see something like the following:

TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State          : enabled
Chimney Offload State               : automatic
NetDMA State                        : enabled
Direct Cache Acess (DCA)            : disabled
Receive Window Auto-Tuning Level    : normal
Add-On Congestion Control Provider  : none
ECN Capability                      : disabled
RFC 1323 Timestamps                 : disabled

I suggest you copy/paste the results to a .txt file to make note of your current settings. The settings you are interested in are Chimney Offload, Receive-Side Scaling (RSS), and NetDMA. These are all features that attempt to offload processing from/to the NIC or CPU, and they can sometimes cause problems with symptoms similar to those that you described.

Before messing with RSS or NetDMA, I would try disabling Chimney Offload by running netsh int tcp set global chimney=disabled and disabling TCP offload under Device Manager > Network Adapters > Advanced tab.

If that doesn't solve your problem, you may need to experiment with the other two options. Here is a Microsoft KB article with details on modifying all of them.