Does Amazon S3 download fail sometimes? [closed]

Question

We just added an autoupdater in our software and got some bug report saying that the autoupdate wouldn't complete properly because the downloaded file's sha1 checksum wasn't matching. We're hosted on Amazon S3...

That's either something wrong with my code or something wrong with S3.

I reread my code for suspicious stuff and wrote a simple script downloading and checking the checksum of the downloaded file, and indeed got a few errors once in while (1 out of 40 yesterday). Today it seems okay.

Did you experience that kind of problem? Is there some kind of workaround ?

extra info: test were ran in Japan.

Answer 1

Amazon's S3 will occasionally fail with errors during uploads or downloads -- generally "500: Internal Server" errors. The error rate is normally pretty low, but it can spike if the service is under heavy load. The error rate is never 0%, so even at the best of times the occasional request will fail.

Are you checking the HTTP response code in your autoupdater? If not, you should check that your download succeeded (HTTP 200) before you perform a checksum. Ideally, your app should retry failed downloads, because transient errors are an unavoidable "feature" of S3 that clients need to deal with.

It is worth noting that if your clients are getting 500 errors, you will probably not see any evidence of these in the S3 server logs. These errors seem to occur before the request reaches the service's logging component.