aws s3 - s3cmd: "WARNING: MD5 signatures do not match:" - what do?

Question

When I use s3cmd to pull down files (of not unreasonable size - less than 100 megabytes) I occasionally see this error: WARNING: MD5 signatures do not match: computed=BLAH, received="NOT-BLAH"

Googling suggests that this may be caused by the way S3 segments files. Others have said to ignore it.

Does anybody know why this happens and what the right thing to do is?

Thank you for your time, -- Henry

Answer 1

Looking into this deeper, it seems as though s3cmd is reading the wrong md5 sum from Amazon. It looks as though s3cmd is getting its sum from the ETAG field. Comparing the actual data of the object that was PUT with the object that was GET'ed the contents are identical and this error can be safely ignored.

Answer 2

The ETag of a file in S3 will not match the MD5 if the file was uploaded as "Multipart". When a file is marked multipart AWS will hash each part, concatenate the results and then hash that value.

If the file does not actually have multiple parts the result will be a hash of a hash with -1 added to the end. Try disabling multipart in the tool you use to upload files to S3. For s3cmd, the option is --disable-multipart.