GCP-Storage: do files appear before upload is complete

Question

I want to transfer files into a VM whenever there is a new file added to storage, the problem is that i want the transfer to be done only when the upload is complete

So my question is : Do files appear even when the upload is still going on ? which means if I build a program that looks for new files every second, would it transfer the files from gcs to VM even if the upload is incomplete or the transfer would start whenever the upload is complete and not while it is uploading ?

Answer 1

Google Cloud Storage uploads are strongly consistent for object uploads. This means that the object is not visible until the object is 100% uploaded and any Cloud Storage housekeeping (such as replication) is complete. You cannot see nor access an object until the upload has completed and your software/tool receives a success response.

Google Cloud Storage Consistency

Do files appear even when the upload is still going on ? which means if I build a program that looks for new files every second, would it transfer the files from gcs to VM even if the upload is incomplete or the transfer would start whenever the upload is complete and not while it is uploading ?

No, your program will not see new objects until the new objects are 100% available. In Google Cloud Storage there are not partial uploads.

Answer 2

Files do not appear in the UI of Cloud Storage until the file is completely upload it to the specified bucket by the user.

I attached you how Google Cloud Platform manage the consistency in Google Cloud Storage Buckets here.

You could use gsutil to list all the files in one of your Cloud Storage Buckets at any moment, as stated here.

As for the application you are trying to develop, I highly suggest you to use Google Cloud Functions, in conjunction with triggers.

In this case, you could use google.storage.object.finalize trigger in order to execute your function every time a new object is uploaded to one of your buckets. You can see examples of this application here.

The Cloud Function will ensure that your bucket is correctly uploaded to the bucket before attempting to transfer the object to your GCE instance.

Therefore, after completing the upload, the only thing left will be to execute gcloud compute scp to copy files to your Google Compute Engine via scp, as stated here.