AWS SDK iOS 2.0 S3 upload: add correct md5

Question

I am trying to upload a file to S3 using the new AWS SDK for iOS 2.0. The upload works fine as long as I don't set a contentMD5 in the request.

First, I create a filepath and a URL:

NSString *tempFilePath = [NSTemporaryDirectory() stringByAppendingPathComponent:@"s3tmp"];
NSURL *tempFileURL = [NSURL fileURLWithPath:tempFilePath];

Next, I create the request:

AWSS3TransferManagerUploadRequest *uploadRequest = [AWSS3TransferManagerUploadRequest new];
uploadRequest.bucket = S3_BUCKETNAME;
uploadRequest.key = s3Key;
uploadRequest.body = tempFileURL;

Then I create the md5. Conveniently, there is a github project here: https://github.com/JoeKun/FileMD5Hash which is creating the md5 from a file. However, I cross checked with bashs md5 and it's returning the same md5 string. Additionally, if I don't set contentMD5 in the request, the upload succeeds and in the console the same md5 string is shown as eTag. So I think the md5 calculated in the next line is correct.

NSString *md5 = [FileHash md5HashOfFileAtPath:tempFilePath];

Finally, I add the md5 to the uploadRequest:

uploadRequest.contentMD5 = md5;

and start the upload:

[[transferManager upload:uploadRequest] continueWithBlock:^id(BFTask *task) {
NSError *error = task.error;
if (error) {
NSDictionary *errorUserInfo = error.userInfo;
NSLog(@"Error %@: %@",[errorUserInfo objectForKey:@"Code"],[errorUserInfo objectForKey:@"Message"]);
dispatch_sync(dispatch_get_main_queue(), ^{
[weakSelf uploadFinishedUnsuccessful];
});
}
else {
NSLog(@"Upload success for file \n%@ to \n%@/%@",[tempFileURL absoluteString],S3_BUCKETNAME,s3Key);
dispatch_sync(dispatch_get_main_queue(), ^{
[weakSelf uploadFinishedSuccessful];
});
}
return nil;
}];

This always returns an error: Error InvalidDigest: The Content-MD5 you specified was invalid.

So I tried wrapping the md5 into base64, using the builtin method of iOS:

NSString *base64EncodedString = [[md5 dataUsingEncoding:NSUTF8StringEncoding] base64EncodedStringWithOptions:0];

I cross checked this with another base64 library. It returns the same base64 string, so I think the base64 string is correct. I tried setting this as contentMD5:

uploadRequest.contentMD5 = base64EncodedString;

I get the same error: Error InvalidDigest: The Content-MD5 you specified was invalid.

Any idea what I am doing wrong?

Thanks for any reply!

Answer 1

You need to base64-encode the binary representation of the MD5 hash... not the hex representation, which is what it sounds like you may be doing.

The resulting value will be somewhere the neighborhood of 24 characters long, if encoded correctly... and twice that long if done incorrectly.

Answer 2

Ok, Michaels comments got me in the right direction. After a lot of reading back and forth I finally understood what he tried to tell me. For anyone else struggling to grasp the concept, I try to explain: An md5 string like "28e01cf6608332ae51d63af3364d77f2" is the hexadecimal representation of a 16 byte digest. That means, each 2 characters of those 32 characters represent 1 byte.

28 = First byte, e0 = Second byte, 1c = third byte, and so on, until you have 16 bytes.

The content-md5 header expects the base64-encoded 16 bytes, not the hexadecimal representation. The otherwise very convenient method

[FileHash md5HashOfFileAtPath:tempFilePath]

returns only this hexadecimal representation as NSString. So I could either dig into extracting the bytes before the string conversion is done, or re-convert the string into NSData. I chose the latter with a code snippet I found here:

//convert the md5 hexadecimal string representation BACK to the NSData byte representation
    NSMutableData *md5Data= [[NSMutableData alloc] init];
    unsigned char whole_byte;
    char byte_chars[3] = {'\0','\0','\0'};
    int i;
    for (i=0; i < [md5 length]/2; i++) {
        byte_chars[0] = [md5 characterAtIndex:i*2];
        byte_chars[1] = [md5 characterAtIndex:i*2+1];
        whole_byte = strtol(byte_chars, NULL, 16);
        [md5Data appendBytes:&whole_byte length:1];
    }

    //base64-encode the NSData byte representation
    NSString *base64EncodedString = [md5Data base64EncodedStringWithOptions:0];

    uploadRequest.contentMD5 = base64EncodedString;

This base64encodedString returns a success response, finally! Thanks Michael!