POST or PUSH operations from s3cmd fail on a certain internet connection

Question

I'm currently automating a build script to push resources to Amazon s3 and am using http://s3tools.org/s3cmd (and https://github.com/s3tools/s3cmd) which I understood was the normal / main command line tool to use. Nothing too complicated, and I had done most of the testing out of office, but as soon as I got into there, the whole thing started to fall apart, and I'm crazy confused why.

A simple command like (with both 'mybucket' existing on s3 and 'file.ext' existing in the directory I'm running the command from),

s3cmd put file.ext s3://mybucket/

was failing with either

[Errno 104] Connection reset by peer

or

[Errno 32] Broken pipe

I know there's an issue with s3 with files over 5GB of size, but these files are nowhere near that, they're less than 1MB, never mind more than 1GB. The really weird thing was that another program, http://www.bucketexplorer.com/ worked perfectly, doing the exact same operations, on the same network.

What was weirder still was to test everything out, I tethered my laptop to my phone's 3G connection, and straight away everything worked perfectly again, and when I got home, and tested the commands again there, it worked perfectly again.

Any idea as to what might be causing this error on our work network, with s3cmd, but not Bucket Explorer?

Answer 1

There can be many reasons for this error, like TCP window scaling (or this one) and DNS propagation. I was able to workaround this by using a tiny multipart chunk size of 5MB:

s3cmd put --multipart-chunk-size=5 file.ext s3://mybucket/