Upload to Amazon S3 with Amazon PHP SDK 2 - MD5 error on same file, everytime

Question

I have a PHP app that uploads videos (from little ones - 1MB - to big ones - 400MB).

Everything works fine, except for some particular files.

These files always presents a MD5 checksum error:

WS Error Code: BadDigest, Status Code: 400, AWS Request ID: 89BBC1D79A4492A7, AWS Error Type: client, AWS Error Message: The Content-MD5 you specified did not match what we received.

I verified the MD5 and it really doesn't mach, but I have no idea why!

If it was a corruption error, the returned S3 MD5 would vary, but it's always the same.

In my local machine (a Mac) and on the server (Ubuntu), the MD5 is:

9131ee88a194b555d0a3519f67294f31

In Amazon, it is:

8e6789baf9c5d434003a5443d30143fa

The upload is made with this excerpt of code:

    try
    {
        $start = (float) array_sum(explode(' ',microtime()));

        $save_path = "/tmp/$video[quality]/$video[video_id].mp4";
        $db_path = "$video[channel]/$video[quality]/$video[video_id].mp4";
        $bytes = number_format(filesize($save_path) / 1048576, 2) . ' MB';

        System_Daemon::info(($i + 1) . "/$num_of_videos - started upload of $video[channel] - $video[video_id] with $bytes");

        $s3 = Aws\S3\S3Client::factory(
                        array(
                            'key' => 'MY KEY',
                            'secret' => 'MY SECRET',
                            'region' => Region::US_EAST_1
                        )
        );


        $results = $s3->putObject(array(
            'Bucket' => 'media.tubelivery.com',
            'Key' => $db_path,
            'Body' => fopen($save_path, 'r'),
            'ACL' => Aws\S3\Enum\CannedAcl::PUBLIC_READ
                ));

        //Delete the original file
        unlink($save_path);
        clearstatcache($save_path);

        //Change the video state to 0
        update_video_state_to_uploaded_to_S3($video['id']);

        $end = (float) array_sum(explode(' ',microtime()));
        $time = sprintf("%.4f", ($end - $start)) . " sec";

        System_Daemon::info("uploaded video " . ($i + 1) . " to $db_path in $time");
    }
    catch (Aws\S3\Exception\S3Exception $e)
    {
        System_Daemon::err("ERROR uploading $video[video_id].mp4 to S3");
        foreach($results as $key => $result)
        {
            System_Daemon::err("$key => $result");
        }
        $save_path = "/tmp/$video[quality]/$video[video_id].mp4";   
        clearstatcache($save_path);
        System_Daemon::err("ERROR: $e");
    }

This is the exact log:

[Mar 13 02:35:53]      err: ERROR uploading 4XwKKMlGibo.mp4 to S3 [l:145]
[Mar 13 02:35:53]      err: Expiration =>  [l:148]
[Mar 13 02:35:53]      err: ServerSideEncryption =>  [l:148]
[Mar 13 02:35:53]      err: ETag => "7f65e3f892d96b9703d411219e2b868a" [l:148]
[Mar 13 02:35:53]      err: VersionId =>  [l:148]
[Mar 13 02:35:53]      err: RequestId => 80821CC621946236 [l:148]
[Mar 13 02:35:53]      err: ERROR: Aws\S3\Exception\BadDigestException: 
                            AWS Error Code: BadDigest, 
                            Status Code: 400, 
                            AWS Request ID: 7C4B4834C6235D1A, 
                            AWS Error Type: client, 
                            AWS Error Message: The Content-MD5 you specified did not match what we received. [l:152]

Any ideas? What am I doing wrong?

Answer 1

I was working with a large data transfer with S3 and had the exact same error message. It turned out that the library we were using made use of the multipart upload functionality for files above a certain size. I'm unsure if the PHP SDK does the same thing, but apparently the way that multipart objects are stored within S3 causes a different hash to be generated than you would get when doing it on your local machine.

I saw that this question was very old, but wanted you to know you were not alone!