Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Does GAE GCS write have asynchronous version like the NDB functions

Does GAE GCS write have asynchronous version like the NDB functions(eg, put_async)?

I found out(from appstats) that since I am uploading multiple files sequentially in my code, it is consuming a lot a time(all of them being sequential). I am trying to reduce this time since with six files it is ~7 seconds and I wish to have more files.

I have put the following code snippet in a for loop which iterates over all the files selected by the user on the webpage:

gcs_file = gcs.open (filename, 'w', content_type = 'image/jpeg') 
gcs_file.write (photo) 
gcs_file.close () 

Please let me know if you need any more data

UPDATE
Original code:

photo_blobkey_list = []             
video_blobkey_list = [] 

i = 0 
for photo in photo_list: 
    filename = bucket + "/user_pic_"+str (user_index) + "_" + str (i) 
    # Store the file in GCS 
    gcs_file = gcs.open (filename, 'w', content_type = 'image/jpeg') 
    gcs_file.write (photo) 
    gcs_file.close () 
    # Store the GCS filename in Blobstore 
    blobstore_filename = '/gs' + filename 
    photo_blobkey = blobstore.create_gs_key (blobstore_filename) 
    photo_blobkey_list.append (photo_blobkey) 
    i = i + 1 

i = 0 
for video in video_list: 
    filename = bucket + "/user_video_"+str (user_index) + "_" + str (i) 
    filename = filename.replace (" ", "_") 
    # Store the file in GCS 
    gcs_file = gcs.open (filename, 'w', content_type = 'video/avi') 
    gcs_file.write (video) 
    gcs_file.close () 
    # Store the GCS filename in Blobstore 
    blobstore_filename = '/gs' + filename 
    video_blobkey = blobstore.create_gs_key (blobstore_filename) 
    video_blobkey_list.append (video_blobkey) 
    i = i + 1 
:: 
user_record.put () 

My Doubt:
Based on the suggestions, I plan to put the GCS write part in a tasklet subroutine which takes filename and the photo/video as argument. How do I use "yield" here so as to make the above code run in parallel as much as possible(all the photo and video write operations)? As per the docs, I need to provide all the arguments to yield so as to make the tasklets run in parallel. But, in my case, the number of photos and videos uploaded is variable number.

Relevant official docs text:

If that were two separate yield statements, they would happen in series. But yielding a tuple of tasklets is a parallel yield: the tasklets can run in parallel and the yield waits for all of them to finish and returns the results.

like image 757
gsinha Avatar asked Oct 18 '25 19:10

gsinha


1 Answers

You can do the write operation in a task.

like image 72
marcadian Avatar answered Oct 20 '25 14:10

marcadian



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!