Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Provide a callback URL in Google Cloud Storage signed URL

When uploading to GCS (Google Cloud Storage) using the BlobStore's createUploadURL function, I can provide a callback together with header data that will be POSTed to the callback URL.

There doesn't seem to be a way to do that with GCS's signed URL's

I know there is Object Change Notification but that won't allow the user to provide upload specific information in the header of a POST, the way it is possible with createUploadURL's callback.

My feeling is, if createUploadURL can do it, there must be a way to do it with signed URL's, but I can't find any documentation on it. I was wondering if anyone may know how createUploadURL achieves that callback calling behavior.


PS: I'm trying to move away from createUploadURL because of the __BlobInfo__ entities it creates, which for my specific use case I do not need, and somehow seem to be indelible and are wasting storage space.


Update: It worked! Here is how:

Short Answer: It cannot be done with PUT, but can be done with POST

Long Answer:

If you look at the signed-URL page, in front of HTTP_Verb, under Description, there is a subtle note that this page is only relevant to GET, HEAD, PUT, and DELETE, but POST is a completely different game. I had missed this, but it turned out to be very important.

There is a whole page of HTTP Headers that does not list an important header that can be used with POST; that header is success_action_redirect, as voscausa correctly answered.

In the POST page Google "strongly recommends" using PUT, unless dealing with form data. However, POST has a few nice features that PUT does not have. They may worry that POST gives us too many strings to hang ourselves with.

But I'd say it is totally worth dropping createUploadURL, and writing your own code to redirect to a callback. Here is how:


Code:

If you are working in Python voscausa's code is very helpful.

I'm using apejs to write javascript in a Java app, so my code looks like this:

            var exp = new Date()
            exp.setTime(exp.getTime() + 1000 * 60 * 100); //100 minutes

            json['GoogleAccessId'] = String(appIdentity.getServiceAccountName())
            json['key'] = keyGenerator()
            json['bucket'] = bucket
            json['Expires'] = exp.toISOString(); 
            json['success_action_redirect'] = "https://" + request.getServerName() + "/test2/";
            json['uri'] = 'https://' + bucket + '.storage.googleapis.com/'; 

            var policy = {'expiration': json.Expires
                        , 'conditions': [
                             ["starts-with", "$key", json.key],
                             {'Expires': json.Expires},
                             {'bucket': json.bucket},
                             {"success_action_redirect": json.success_action_redirect}
                           ]
                        };

            var plain = StringToBytes(JSON.stringify(policy))
            json['policy'] = String(Base64.encodeBase64String(plain))
            var result = appIdentity.signForApp(Base64.encodeBase64(plain, false));
            json['signature'] = String(Base64.encodeBase64String(result.getSignature()))

The code above first provides the relevant fields. Then creates a policy object. Then it stringify's the object and converts it into a byte array (you can use .getBytes in Java. I had to write a function for javascript). A base64 encoded version of this array, populates the policy field. Then it is signed using the appidentity package. Finally the signature is base64 encoded, and we are done.

On the client side, all members of the json object will be added to the Form, except the uri which is the form's address.

        var formData = new FormData(document.forms.namedItem('upload'));
        var blob = new Blob([thedata], {type: 'application/json'})
        var keys = ['GoogleAccessId', 'key', 'bucket', 'Expires', 'success_action_redirect', 'policy', 'signature']
        for(field in keys)
          formData.append(keys[field], url[keys[field]])
        formData.append('file', blob)
        var rest = new XMLHttpRequest();
        rest.open('POST', url.uri)
        rest.onload = callback_function
        rest.send(formData)

If you do not provide a redirect, the response status will be 204 for success. But if you do redirect, the status will be 200. If you got 403 or 400 something about the signature or policy maybe wrong. Look at the responseText. If is often helpful.


A few things to note:

  • Both POST and PUT have a signature field, but these mean slightly different things. In case of POST, this is a signature of the policy.
  • PUT has a baseurl which contains the key (object name), but the URL used for POST may only include bucket name
  • PUT requires expiration as seconds from UNIX epoch, but POST wants it as an ISO string.
  • A PUT signature should be URL encoded (Java: by wrapping it with a URLEncoder.encode call). But for POST, Base64 encoding suffices.
  • By extension, for POST do Base64.encodeBase64String(result.getSignature()), and do not use the Base64.encodeBase64URLSafeString function
  • You cannot pass extra headers with the POST; only those listed in the POST page are allowed.
  • If you provide a URL for success_action_redirect, it will receive a GET with the key, bucket and eTag.
  • The other benefit of using POST is you can provide size limits. With PUT however, if a file breached your size restriction, you can only delete it after it was fully uploaded, even if it is multiple-tera-bytes.

What is wrong with createUploadURL?

The method above is a manual createUploadURL. But:

  • You don't get those __BlobInfo__ objects which create many indexes and are indelible. This irritates me as it wastes a lot of space (which reminds me of a separate issue: issue 4231. Please go give it a star)
  • You can provide your own object name, which helps create folders in your bucket.
  • You can provide different expiration dates for each link.

For the very very few javascript app-engineers:

function StringToBytes(sz) {
  map = function(x) {return x.charCodeAt(0)}
  return sz.split('').map(map)
}
like image 725
user2782503 Avatar asked Jan 03 '16 03:01

user2782503


People also ask

How do I get a signed URL for Google Cloud Storage?

Options for generating a signed URL Simply specify Cloud Storage resources, point to the host storage.googleapis.com , and use Google HMAC credentials in the process of generating the signed URL.

What is signed URL in GCP?

Signed URLs give time-limited resource access to anyone in possession of the URL, regardless of whether the user has a Google Account. A signed URL is a URL that provides limited permission and time to make a request.

How do I create a signed URL?

A signed URL will be produced for each provided URL, authorized for the specified HTTP method and valid for the given duration. The signurl command uses the private key for a service account (the '<private-key-file>' argument) to generate the cryptographic signature for the generated URL.

What is the difference between CloudFront signed URL and S3 signed URL?

Both S3 and CloudFront have URL signing features that work differently. However, only S3 refers to them as Pre-signed URLs; CloudFront refers to them as Signed URLs and Signed Cookies. Note the service names in the URLs, in the documentation below.


2 Answers

You can include succes_action_redirect in a policy document when you use GCS post object.

Docs here: Docs: https://cloud.google.com/storage/docs/xml-api/post-object
Python example here: https://github.com/voscausa/appengine-gcs-upload

Example callback result:

def ok(self):
    """ GCS upload success callback """

    logging.debug('GCS upload result : %s' % self.request.query_string)
    bucket = self.request.get('bucket', default_value='')
    key = self.request.get('key', default_value='')
    key_parts = key.rsplit('/', 1)
    folder = key_parts[0] if len(key_parts) > 1 else None
like image 129
voscausa Avatar answered Oct 08 '22 00:10

voscausa


A solution I am using is to turn on Object Changed Notifications. Any time an object is added, a Post is sent to a URL - in my case - a servlet in my project.

In the doPost() I get all info of objected added to GCS and from there, I can do whatever.

This worked great in my App Engine project.

like image 26
Micro Avatar answered Oct 08 '22 02:10

Micro