AWS S3 pre-signed URL with enforced content-md5

Question

I'm trying to use S3´s pre-signed URLs with an enforced Content-MD5. Therefore I'm basically trying to follow the example of their Docs. Obviously I'm doing something wrong.

Here is the checksum of the file I try to upload:

➜ md5 testfile.txt
MD5 (testfile.txt) = ce0a4a83c88c2e7562968f03076ae62f

Here is the code:

func main() {

    sess, err := session.NewSession(&aws.Config{
        Region: aws.String("eu-central-1")},
    )

    svc := s3.New(sess)

    resp, _ := svc.PutObjectRequest(&s3.PutObjectInput{
        Bucket: aws.String("bucket"),
        Key:    aws.String("testfile.txt"),
    })

    md5 := "ce0a4a83c88c2e7562968f03076ae62f" // hard coded & pasted from "$ md5 testfile.txt"
    md5s := base64.StdEncoding.EncodeToString([]byte(md5))

    resp.HTTPRequest.Header.Set("Content-MD5", md5s)

    url, err := resp.Presign(15 * time.Minute)
    if err != nil {
        fmt.Println("error presigning request", err)
        return
    }

    fmt.Printf("curl -XPUT -H \"Content-MD5: %s\" %s --upload-file %s\n\n", md5s, url, "testfile.txt")
}

Which should give me a ready-to-use curl command like: curl -XPUT -H "Content-MD5: Y2UwYTRhODNjODhjMmU3NTYyOTY4ZjAzMDc2YWU2MmY=" https://bucket.s3.eu-central-1.amazonaws.com/testfile.txt<super-long-url> --upload-file testfile.txt

Unfortunately the request always fails with this message:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidDigest</Code><Message>The Content-MD5 you specified was invalid.</Message><Content-MD5>Y2UwYTRhODNjODhjMmU3NTYyOTY4ZjAzMDc2YWU2MmY=</Content-MD5><RequestId>24F73D8948824799</RequestId><HostId>uKgSjxi03P4EvBk+Yo/EzxqWT0AI6AN3FPB2bKKAtgVjp8t4q2Ku+Tvui108vIQgcwgfvQdwmrk=</HostId></Error>

As I was a bit unsure whether I should request with the base64 of the MD5 I tried it with the normal MD5 as well which responses with

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId><accesskeyid></AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20180127T215418Z
20180127/eu-central-1/s3/aws4_request
e9580e510332d2fe8811209a8952e849022a56b93a02eca037fa43a10dec680f</StringToSign><SignatureProvided><signature></SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 38 30 31 32 37 54 32 31 35 34 31 38 5a 0a 32 30 31 38 30 [...]
</StringToSignBytes><CanonicalRequest>PUT
/testfile.txt
X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&amp;X-Amz-Credential=<accesskeyid>%2F20180127%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20180127T215418Z&amp;X-Amz-Expires=900&amp;X-Amz-SignedHeaders=content-md5%3Bhost
content-md5:ce0a4a83c88c2e7562968f03076ae62f
host:bucket.s3.eu-central-1.amazonaws.com

content-md5;host
UNSIGNED-PAYLOAD</CanonicalRequest><CanonicalRequestBytes>50 55 54 0a 2f 74 65 73 74 66 69 6c 65 2e 74 78 74 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 [...]
] 44</CanonicalRequestBytes><RequestId>D92C97EE37BE602A</RequestId><HostId>VatR9cidZlUgq+Ngd5vkZ+wHNiumsCPhx/TvZnwImAkj/STZ0eXazVrwGPRdketBbICd91VLG9E=</HostId></Error>

An upload works as soon as I remove the header settingresp.HTTPRequest.Header.Set("Content-MD5", md5s) and request with curl -XPUT https://bucket.s3.eu-central-1.amazonaws.com/testfile.txt<super-long-url> --upload-file testfile.txt.

What am I doing wrong?

Answer 1

Because of the way base64 encoding works, the base64 representation of an md5 will always be exactly 24 characters long and the last 2 characters will always be ==. As you see, yours is about twice as long as it should be.

An actual md5 digest/hash is only 16 bytes (128 bits) long, and is a non-printable binary blob.

The md5sum utility and similar tools return the digest in a hex-encoded, printable format, which is 32 bytes long, consisting of only the characters 0-9 and a-f... it's the same value, but it's already been passed through hex-encoding, so that isn't the representation that you need to start with, if you want to base64-encode the md5 as required in the Content-MD5 header.

openssl dgst -md5 -binary {filename} will generate the binary representation of the md5 of the file, or you can use a pipe to actually generate the final base-64 representation with openssl dgst -md5 -binary {filename} | base64.

Note that this has nothing to do with ssl of course, but I used the openssl dgst tool for this example because it's probably something you already happen to have on your system, as well as the base64 conversion tool, which is probably already there, too.