PUT/Copy With PHP, REST, Flex, and Amazon S3

Question

I have been trying for weeks to properly format a REST request to the Amazon AWS S3 API using the available examples on the web but have been unable to even successfully connect.

I have found the code to generate a signature, found the proper method for formatting the "string to encode", and the http headers. I have worked my way through the signatureDoesNotMatch errors just to get a Anonymous users can not perform copy functions, Please authenticate message.

I have a working copy of an Adobe Flex application that successfully uploads files, but with their "original" file name. The point of using the REST with the Amazon API is to perform a PUT (copy) of the file, just so I can rename it to something my back end system can use.

If I could find a way to get this REST submission to work, or perhaps a way to specify a "new" filename within Flex while uploading I could avoid this whole REST situation all together.

If anyone has successfully performed a PUT/Copy command on the Amazon API via REST I would be very interested in how this was accomplished - OR - if anyone has been able to change the destination file name using the Flex fileReference.browse() method I would also be eternally grateful for any pointers.

---

PHP code for this is as follows:

$aws_key = 'removed_for_security';
$aws_secret = 'removed_for_security';
$source_file = $uploaded_s3_file; // file to upload to S3 (defined in above script)
$aws_bucket = 'bucket'; // AWS bucket
$aws_object = $event_file_name; // AWS object name (file name)
if (strlen($aws_secret) != 40) die("$aws_secret should be exactly 40 bytes long");
$file_data = file_get_contents($source_file);
if ($file_data == false) die("Failed to read file " . $source_file);

// opening HTTP connection to Amazon S3
$fp = fsockopen("s3.amazonaws.com", 80, $errno, $errstr, 30);
if (!$fp) die("$errstr ($errno)\n");

// Uploading object
$file_length = strlen($file_data); // for Content-Length HTTP field
$dt = gmdate('r'); // GMT based timestamp

// preparing String to Sign (see AWS S3 Developer Guide)
// preparing string to sign
$string2sign = "PUT


{$dt}
/{$aws_bucket}/{$aws_object}";

// preparing HTTP query 
// $query = "PUT /".$aws_bucket."/".$event_file_name." HTTP/1.1
$query = "PUT /" . $event_file_name . " HTTP/1.1
Host: {$aws_bucket}.s3.amazonaws.com
Date: {$dt}
x-amz-copy-source: /{$aws_bucket}/{$current_s3_filename}
x-amz-acl: public-read

Authorization: AWS {$aws_key}:" . amazon_hmac($string2sign) . "\n\n";

$query .= $file_data;
$resp = sendREST($fp, $query);
if (strpos($resp, '') !== false) {
     die($resp);
}
echo "FILE uploaded\n";

// done
echo "Your file's URL is: http://s3.amazonaws.com/{$aws_bucket}/{$aws_object}\n";
fclose($fp);

// Sending HTTP query and receiving, with trivial keep-alive support
function sendREST($fp, $q, $debug = true){
     if ($debug) echo "\nQUERY<<{$q}>>\n";
     fwrite($fp, $q);
     $r = '';
     $check_header = true;
     while (!feof($fp)) {
          $tr = fgets($fp, 256);
          if ($debug) echo "\nRESPONSE<<{$tr}>>";
          $r .= $tr;
          if (($check_header) && (strpos($r, "\r\n\r\n") !== false)) {

               // if content-length == 0, return query result
               if (strpos($r, 'Content-Length: 0') !== false) {
                    return $r;
               }
          }

          // Keep-alive responses does not return EOF
          // they end with \r\n0\r\n\r\n string
          if (substr($r, -7) == "\r\n0\r\n\r\n") {
               return $r;
          }
     }
     return $r;
}

// hmac-sha1 code START
// hmac-sha1 function: assuming key is global $aws_secret 40 bytes long
// read more at http://en.wikipedia.org/wiki/HMAC
// warning: key($aws_secret) is padded to 64 bytes with 0x0 after first function call
function amazon_hmac($stringToSign) {

     // helper function binsha1 for amazon_hmac (returns binary value of sha1 hash)
     if (!function_exists('binsha1')) {
          if (version_compare(phpversion(), "5.0.0", ">=")) {
               function binsha1($d) { return sha1($d, true); }
          } else {
               function binsha1($d) { return pack('H*', sha1($d)); }
          }
     }
     global $aws_secret;
     if (strlen($aws_secret) == 40) {
          $aws_secret = $aws_secret . str_repeat(chr(0), 24);
     }
     $ipad = str_repeat(chr(0x36), 64);
     $opad = str_repeat(chr(0x5c), 64);
     $hmac = binsha1(($aws_secret ^ $opad) . binsha1(($aws_secret ^ $ipad) . $stringToSign));
     return base64_encode($hmac);
}
// hmac-sha1 code END

When I submit a malformed or incorrect header I get the corresponding error message as expected:

Query:

PUT /bucket/1-132-1301047200-1.jpg HTTP/1.1 Host: s3.amazonaws.com x-amz-acl: public-read Connection: keep-alive Content-Length: 34102 Date: Sat, 26 Mar 2011 00:43:36 +0000 Authorization: AWS -removed for security-:GmgRObHEFuirWPwaqRgdKiQK/EQ=

HTTP/1.1 403 Forbidden
x-amz-request-id: A7CB0311812CD721
x-amz-id-2: ZUY0mH4Q20Izgt/9BNhpJl9OoOCp59DKxlH2JJ6K+sksyxI8lFtmJrJOk1imxM/A
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Sat, 26 Mar 2011 00:43:36 GMT
Connection: close
Server: AmazonS3
397 SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your key and signing method.50 55 54 0a 0a 0a 53 61 74 2c 20 32 36 20 4d 61 72 20 32 30 31 31 20 30 30 3a 34 33 3a 33 36 20 2b 30 30 30 30 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 70 75 62 6c 69 63 2d 72 65 61 64 0a 2f 6d 6c 68 2d 70 72 6f 64 75 63 74 69 6f 6e 2f 31 2d 31 33 32 2d 31 33 30 31 30 34 37 32 30 30 2d 31 2e 6a 70 67A7CB0311812CD721ZUY0mH4Q20Izgt/9BNhpJl9OoOCp59DKxlH2JJ6K+sksyxI8lFtmJrJOk1imxM/AGmgRObHEFuirWPwaqRgdKiQK/EQ=PUT Sat, 26 Mar 2011 00:43:36 +0000 x-amz-acl:public-read /bucket/1-132-1301047200-1.jpg-removed for security- 0

but when sending properly formatted requests, it says I'm not authenticated:

Query being used:

PUT /1-132-1301047200-1.jpg HTTP/1.1 Host: bucket.s3.amazonaws.com Date: Sat, 26 Mar 2011 00:41:50 +0000 x-amz-copy-source: /bucket/clock.jpg x-amz-acl: public-read Authorization: AWS -removed for security-:BMiGhgbFnVAJyiderKjn1cT7cj4=

HTTP/1.1 403 Forbidden
x-amz-request-id: ABE45FD4DFD19927
x-amz-id-2: CnkMmoF550H1zBlrwwKfN8zoOSt7r/zud8mRuLqzzBrdGguotcvrpZ3aU4HR4RoO
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Sat, 26 Mar 2011 00:41:50 GMT
Server: AmazonS3

AccessDenied
Anonymous users cannot copy objects. Please authenticate
ABE45FD4DFD19927CnkMmoF550H1zBlrwwKfN8zoOSt7r/zud8mRuLqzzBrdGguotcvrpZ3aU4HR4RoO 0
Date: Sat, 26 Mar 2011 00:41:50 GMT
Connection: close
Server: AmazonS3

Answer 1

I have been trying for weeks to properly format a REST request to the Amazon AWS S3 API using the available examples on the web

Have you tried the Amazon AWS SDK for PHP? It's comprehensive, complete, and most importantly, written by Amazon. If their own code isn't working for you, something's gonna be really wrong.

---

Here is example code using the linked SDK to upload example.txt in the current directory to a bucket named 'my_very_first_bucket'.

<?php
// Complain wildly.
    ini_set('display_errors', true);
    error_reporting(-1);
// Set these yourself.
    define('AWS_KEY', '');
    define('AWS_SECRET_KEY', '');
// We'll assume that the SDK is in our current directory
    include_once 'sdk-1.3.1/sdk.class.php';
    include_once 'sdk-1.3.1/services/s3.class.php';
// Set the bucket and name of the file we're sending.
// It happens that we're actually uploading the file and 
// keeping the name, so we're re-using the variable
// below.
    $bucket_name = 'my_very_first_bucket';
    $file_to_upload = 'example.txt';
// Fire up the object
    $s3 = new AmazonS3(AWS_KEY, AWS_SECRET_KEY);
// This returns a "CFResponse"
    $r = $s3->create_object(
        $bucket_name,
        $file_to_upload,
        array(
        // Filename of the thing we're uploading
            'fileUpload' => (__DIR__ . '/' . $file_to_upload),
        // ACL'd public.
            'acl' => AmazonS3::ACL_PUBLIC,
        // No wai.
            'contentType' => 'text/plain',
        // The docs say it'll guess this, but may as well.
            'length' => filesize(__DIR__ . '/' . $file_to_upload)
        )
    );
// Did it work?
    echo "Worked: ";
    var_dump($r->isOK());
// Status as in HTTP.
    echo "\nStatus: ";
    var_dump($r->status);
// The public URL by which we can reach this object.
    echo "\nURL: ";
    echo $s3->get_object_url($bucket_name, $file_to_upload);
// Tada!
    echo "\n";

Appropriate API docs:

• get_object_url
• create_object
• The CFResponse class.

You can navigate the rest of the methods in the left menu. It's pretty comprehensive, including new bucket creation, management, deletion, same for objects, etc.

You should be able to basically drop this in to your code and have it work properly. PHP 5.2-safe.

---

Edit by Silver Tiger:

Charles -

     The method you provide is using the API SDK functions to upload a file from the local file system to a bucket of my choosing. I have that part working already via Flex and uploads work like a charm. The problem in question is being able to submit a REST request to AWS S3 to change the file name from it's current "uploaded" name, to a new name more suited name that will work with my back end (database, tracking etc, which I handle and display seperately in PHP with MyySQL).

     AWS S3 does not truly support a "copy" function, so they provided a method to re-"PUT" a file by reading the source from your own bucket and placing a new copy with a different name in the same bucket. The difficulty I have been having is processing the REST request, hence the HMAC encryption.

     I do appreciate your time and understand the example you have provided as i also have a working copy of the PHP upload that was functioning before I designed the Flex application. The reason for the Flex was to enable status updates and a dynamically updated progress bar, which is also working like a charm :).

I will continue to persue a REST solution as from the perspective of Amason zupport, it will be the only way i can rename a file already existing in my bucket per thier support team.

As always, if you have input or suggestions regarding the REST submission I would be greatful for any feedback.

Thanks,

Silver Tiger

---

Proof copy/delete works:

    $r = $s3->copy_object(
        array( 'bucket' => $bucket_name, 'filename' => $file_to_upload ),
        array( 'bucket' => $bucket_name, 'filename' => 'foo.txt' )
    );
// Did it work?
    echo "Worked: ";
    var_dump($r->isOK());
// Status as in HTTP.
    echo "\nStatus: ";
    var_dump($r->status);

// The public URL by which we can reach this object.
    echo "\nURL: ";
    echo $s3->get_object_url($bucket_name, 'foo.txt');

    echo "\nDelete: ";
// Nuke?
    $r = $s3->delete_object($bucket_name, $file_to_upload);
// Did it work?
    echo "Worked: ";
    var_dump($r->isOK());
// Status as in HTTP.
    echo "\nStatus: ";
    var_dump($r->status);

---

Edit by Silver Tiger:

Charles -

     No REST needed, no bothers ... SDK 1.3.1 and your help solved the issue. the code I used to test looks a lot like yours :

// Complain wildly.
    ini_set('display_errors', true);
    error_reporting(-1);
// Set these yourself.
    define('AWS_KEY', 'removed for security');
    define('AWS_SECRET_KEY', 'removed for security');
// We'll assume that the SDK is in our current directory
    include_once 'includes/sdk-1.3.1/sdk.class.php';
    include_once 'includes/sdk-1.3.1/services/s3.class.php';
// Set the bucket and name of the file we're sending.
// It happens that we're actually uploading the file and 
// keeping the name, so we're re-using the variable
// below.
    $bucket = 'bucket';
    $file_to_upload = 'example.txt';
    $Source_file_to_copy = 'Album.jpg';
    $Destination_file = 'Album2.jpg';
// Fire up the object
// Instantiate the class
$s3 = new AmazonS3();
$response = $s3->copy_object(
    array( // Source
        'bucket' => $bucket,
        'filename' => $Source_file_to_copy
    ),
    array( // Destination
        'bucket' => $bucket,
        'filename' => $Destination_file
    )
);
// Success?
var_dump($response->isOK());

Now I will implement the delete after the copy, and we're golden. Thank you sir for your insight and help.

Silver Tiger