Why doesn't Google Storage API work with Amazon SDK v3?

Question

I feel pretty odd posting this here but since SO is the only official channel for Google API support, I guess I need to ask it here.

If you have API, tool usage, or other software development-related questions, search for and post questions on Stack Overflow, using the official google-cloud-storage tag.

Ok, so here goes. I've spent the better part of two days working on trying to get Google Storage to work on the v3 version (latest) of Amazon's PHP SDK. I can't use an older version of the SDK because I'm trying to stick to Laravel 5.1's filesystem without having to write a brand new driver for Google Storage. I believe this is within the spirit of what Google advertises for Google Storage:

https://cloud.google.com/storage/docs/migrating

In a simple migration from Amazon S3 to Google Cloud Storage, you can use your existing tools and libraries for generating authenticated REST requests to Amazon S3, to also send authenticated requests to Google Cloud Storage. The changes you need to make to your existing tools and libraries are described in this section.

To get set up for a simple migration do the following:

Set a default Google project.

Get a developer key.

In your existing tools or libraries, make the following changes: Change the request endpoint to use the Google Cloud Storage request endpoint. Replace the Amazon Web Services (AWS) access and secret key with the corresponding Google Cloud Storage access key and secret key (collectively called your Google developer key).

That's it! At this point you can start using your existing tools and libraries to send keyed-hash message authentication code (HMAC) requests to Google Cloud Storage.

What a pitch! Let's give it a try using Interoperability credentials that work using gsutil.

$client = new S3Client([
    'credentials' => [
        'key' => 'GOOGxxxxxxxxxxxxxxx',
        'secret' => 'ZfcOTxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
    ],
    'region' => 'US',
    'version' => 'latest',
    'endpoint' => 'https://storage.googleapis.com',
]);

try {
    $result = $client->putObject(array(
        'Bucket' => 'devtest',
        'Key' => 'test',
        'Body' => 'Hello world'
    ));

    echo $result['ObjectURL'];
} catch (\Aws\S3\Exception\S3Exception $e) {
    // The AWS error code (e.g., )
    echo $e->getAwsErrorCode() . "\n";
    // The bucket couldn't be created
    echo $e->getMessage() . "\n";
}

Doesn't work. You get an "Incorrect Authentication Header". Let's take a look at that header.

AWS4-HMAC-SHA256 Credential=GOOGGUxxxxxxxxxxx/20150611/US/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=9c7de4xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

I created a SO post about this at this point, and someone suggested that I add 'signature' => 'v2'.

Google Storage Incorrect Authorization Header with Amazon S3 PHP SDK v3

Let's try that:

$client = new S3Client([
    'credentials' => [
        'key' => 'GOOGxxxxxxxxxxxxxxx',
        'secret' => 'ZfcOTxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
    ],
    'region' => 'US',
    'version' => 'latest',
    'endpoint' => 'https://storage.googleapis.com',
    'signature' => 'v2',
]);

No luck. Same error. The authorization header hasn't changed. Let's look at S3Client's code and see how 'signature' gets used:

public function createPresignedRequest(CommandInterface $command, $expires)
{
    /** @var \Aws\Signature\SignatureInterface $signer */
    $signer = call_user_func(
        $this->getSignatureProvider(),
        $this->getConfig('signature_version'),
        $this->getApi()->getSigningName(),
        $this->getRegion()
    );

    return $signer->presign(
        \Aws\serialize($command),
        $this->getCredentials()->wait(),
        $expires
    );
}

It doesn't. So now we're deviating from S3's official documentation because they say the same thing:

http://docs.aws.amazon.com/aws-sdk-php/guide/latest/configuration.html

It's not 'signature', it's 'signature_version'. Let's change that to v2.

$client = new S3Client([
    'credentials' => [
        'key' => 'GOOGxxxxxxxxxxxxxxx',
        'secret' => 'ZfcOTxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
    ],
    'region' => 'US',
    'version' => 'latest',
    'endpoint' => 'https://storage.googleapis.com',
    'signature_version' => 'v2',
]);

At least we get a different error this time!

UnresolvedSignatureException in SignatureProvider.php line 61: Unable to resolve a signature for v2/s3/US. Valid signature versions include v4 and anonymous.

So, after toying with this for two days it looks like it isn't possible, at least not with the ease that Google wants you to believe in their pitch. I can't get this to work at all, so I'm hoping someone here can shed some light on this. Either I missed something important, or Google is falsely advertising that Google Storage works using Amazon's S3 SDK and wasting developers' time. I'm thinking that maybe we have to manually hijack the authorization header, but that's outside of my expertise. Any help would be greatly appreciated.

Answer 1

So I'm going to post later than never and say that I'm using laravel 5.1, you have to make a choice on whether you want to go with AWS or GCS at the time of configuration as you can't install two versions of the aws php client (which limits your choice of flysystem to v2 or v3. GCS requires v2 for the time being while the s3 implementation in laravel requires v3. I spent hours trying to get v3 to work with GCS but the authentication headers are different so I didn't bother.

You do have to provide your own provider, but it's not that difficult to setup. Just create GcsAppsServiceProvider.php in app/Providers

<?php namespace App\Providers;

use Illuminate\Support\ServiceProvider;
use Storage;
use Aws\S3\S3Client;
use League\Flysystem\AwsS3v2\AwsS3Adapter;
use League\Flysystem\Filesystem;

class GcsAppsServiceProvider extends ServiceProvider
{
    public function register()
    {

    }

    public function boot()
    {
        Storage::extend('gcs', function( $app, $config )
        {
            $client = S3Client::factory([
                'key'    => $config['key'],
                'secret' => $config['secret'],
                'base_url' => $config['base_url'],
            ]);

            return new Filesystem(new AwsS3Adapter($client, $config['bucket']));
        });
    }
}

Essentially in your config you're just duplicating the s3 config in filesystem.php and changing region -> base_url. You're also free to change the default and cloud to gcs to support google cloud storage.

's3' => [
    'driver' => 's3',
    'key'    => env('AWS_KEY', 'your-key'),
    'secret' => env('AWS_SECRET', 'your-secret'),
    'region' => env('AWS_REGION','your-region'),
    'bucket' => env('AWS_BUCKET','your-bucket'),
],

'gcs' => [
    'driver' => 'gcs',
    'key'    => env('GCS_KEY', 'your-key'),
    'secret' => env('GCS_SECRET', 'your-secret'),
    'base_url' => env('GCS_BASE_URL','your-base-url'),
    'bucket' => env('GCS_BUCKET','your-bucket'),
],

Last but not least you also need to add the provider to the provider list in app.php

'providers' => [
    ...
    App\Providers\GcsAppsServiceProvider::class,
],

The AWS S3 adapter also mentions that GCS supports it in their documentation.