Does GAE GCS write have asynchronous version like the NDB functions(eg, put_async)?
I found out(from appstats) that since I am uploading multiple files sequentially in my code, it is consuming a lot a time(all of them being sequential). I am trying to reduce this time since with six files it is ~7 seconds and I wish to have more files.
I have put the following code snippet in a for loop which iterates over all the files selected by the user on the webpage:
gcs_file = gcs.open (filename, 'w', content_type = 'image/jpeg')
gcs_file.write (photo)
gcs_file.close ()
Please let me know if you need any more data
UPDATE
Original code:
photo_blobkey_list = []
video_blobkey_list = []
i = 0
for photo in photo_list:
filename = bucket + "/user_pic_"+str (user_index) + "_" + str (i)
# Store the file in GCS
gcs_file = gcs.open (filename, 'w', content_type = 'image/jpeg')
gcs_file.write (photo)
gcs_file.close ()
# Store the GCS filename in Blobstore
blobstore_filename = '/gs' + filename
photo_blobkey = blobstore.create_gs_key (blobstore_filename)
photo_blobkey_list.append (photo_blobkey)
i = i + 1
i = 0
for video in video_list:
filename = bucket + "/user_video_"+str (user_index) + "_" + str (i)
filename = filename.replace (" ", "_")
# Store the file in GCS
gcs_file = gcs.open (filename, 'w', content_type = 'video/avi')
gcs_file.write (video)
gcs_file.close ()
# Store the GCS filename in Blobstore
blobstore_filename = '/gs' + filename
video_blobkey = blobstore.create_gs_key (blobstore_filename)
video_blobkey_list.append (video_blobkey)
i = i + 1
::
user_record.put ()
My Doubt:
Based on the suggestions, I plan to put the GCS write part in a tasklet subroutine which takes filename and the photo/video as argument. How do I use "yield" here so as to make the above code run in parallel as much as possible(all the photo and video write operations)? As per the docs, I need to provide all the arguments to yield so as to make the tasklets run in parallel. But, in my case, the number of photos and videos uploaded is variable number.
Relevant official docs text:
If that were two separate yield statements, they would happen in series. But yielding a tuple of tasklets is a parallel yield: the tasklets can run in parallel and the yield waits for all of them to finish and returns the results.
You can do the write operation in a task.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With