Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Google Cloud Storage - GSUtil - Copy files, skip existing, do not overwrite

Tags:

I want to sync a local directory to a bucket in Google Cloud Storage. I want to copy the local files that do not exist remotely, skipping files that already exist both remote and local. Is this possible to do this with GSUtil? I cant seem to find a "sync" option for GSUtil or a "do not overwrite". Is it possible to script this?

I am on Linux (Ubuntu 12.04)?

like image 304
ECII Avatar asked Jul 21 '12 10:07

ECII


People also ask

What does gsutil rsync do?

The gsutil rsync command copies changed files in their entirety and does not employ the rsync delta-transfer algorithm to transfer portions of a changed file.

Does gsutil CP create folder?

You cannot create folders with gsutil as gsutil does not support it (workaround see below). However, it is supported via: UI in browser. write your own GCS client (we have written our own custom client which can create folders)


2 Answers

gsutil now supports the noclobber flag (-n) on the cp command. Update your gsutil to the latest version (using gsutil update) and then use the -n flag when performing a copy.

This flag will skip files that already exist at the destination.

like image 147
rein Avatar answered Sep 20 '22 12:09

rein


You need to add (-n) to the command, mentioned officially on Google Cloud Platform:

-n: No-clobber. When specified, existing files or objects at the destination will not be overwritten. Any items that are skipped by this option will be reported as being skipped. This option will perform an additional GET request to check if an item exists before attempting to upload the data. This will save retransmitting data, but the additional HTTP requests may make small object transfers slower and more expensive.

Example (Using multithreading):

gsutil -m cp -n -a public-read -R large_folder gs://bucket_name
like image 22
My Name Avatar answered Sep 16 '22 12:09

My Name