Upload files to Google cloud storage from appengine app

Question

I'm sure the answer to this question is easy, but for me it's proven to be very frustrating since I can't put any solution I've found into practical code for my own use.

I'm building an app on the app engine that let's the user upload a file, which then gets acted on by the app. The size of the file is typically around a few Mbs, and in some cases upto maybe 20 Mb or so. This is enough to trigger the 30 second timeout in app engine, and I am therefore trying to upload to cloud storage directly as suggested on various websites (including here).

The upload part of my form looks like this:

<form name="convert" action="/transform" method="post" enctype="multipart/form-data">
    <input type="file" name="coords">
    <input type="submit" name="submit" value="Transform">
</form>

And the handler looks like this (I'm only starting out with the app engine, so apologies if this looks stupid.

class TransformPage(BaseHandler):
    def get(self):
    self.render_template('transform.html',name=self.request.get('form'))

    def post(self):     
        filename = self.request.POST['coords'].filename
        filesize = int(self.request.headers['Content_Length'])

        if filesize>5242880:
            self.redirect('/error')
        else:
             save_to_cloud(file.value,filename)
             # Start processing file here

I put in the check for file size as a very simple way of preventing the 30s timeout, and 5Mb seems to be the maximum size I can push through in 30s on my network.

The save_to_cloud() is just a wrapper I wrote around the Google cloud storage API, and looks like this:

def save_to_cloud(f,filename):
    filename = BUCKET + filename                                                          
    create_file_and_write(f,filename)

The above all works, and I see the uploaded files ending up in cloud storage, but as I said above, breaks down for larger files. I've seen hints about using create_upload_url searching around, but I am not good enough to see how that pertains to my case, in particular getting the contents of the uploaded file read in to memory so that I can write it to cloud storage.

This is probably the easiest thing in the world if you know how it's done, but I can't do it without someone showing me in code how it can be done :-(

Thanks

Edit: @Itamar: this is indeed what I am trying to accomplish, although the upload form also contains a few other selections to be made by the user. I have now changed my code to this

class TransformPage(BaseHandler):
    def get(self):
        upload_url =blobstore.create_upload_url('/upload',gs_bucket_name='/my_bucket/')


        self.render_template('transform.html',
                            {'name':self.request.get('form'),                 
                            {'upload_url':upload_url})

But I can't get the upload_url to show up in the template html which looks like this

<form name="convert" action="{{ upload_url }}" method="post" enctype="multipart/form-data">

Answer 1

If I understand correctly, what you're trying to do is serve a form from App Engine that allows a user to choose a file to upload. Since the uploaded file may be large, you don't want to handle the upload in App Engine, but have the file uploaded directly to Google Cloud Storage.

This can be done, and it's not too difficult. In fact, this is exactly the example given in the App Engine Python Blobstore docs. It might be confusing that the example refers to Blobstore while you want the file in Cloud Storage - but it's OK - it appears that since version 1.7.0 you can do this:

upload_url = blobstore.create_upload_url('/upload', gs_bucket_name='my_bucket')

(instead of step 1 in the example I linked), and the upload URL will upload straight to Cloud Storage.

Now, your form action should be the string upload_url that was returned by the blobstore.create_upload_url function.

After the form completes processing (meaning the file is uploaded), it will redirect to the argument you passed to that function (in the example- to /upload).

You do not need to write your own POST processing function, as you described in the question.