Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Gsutil - How can I check if a file exists in a GCS bucket (a sub-directory) using Gsutil

I have a GCS bucket containing some files in the path

gs://main-bucket/sub-directory-bucket/object1.gz

I would like to programmatically check if the sub-directory bucket contains one specific file. I would like to do this using gsutil.

How could this be done?

like image 996
activelearner Avatar asked Mar 30 '15 22:03

activelearner


5 Answers

You can use the gsutil stat command.

like image 125
jterrace Avatar answered Nov 20 '22 09:11

jterrace


Use the gsutil stat command. For accessing the sub-directories with more number of files use wildcards(*).

For example:

gsutil -q stat gs://some-bucket/some-subdir/*; echo $?

In your case:

gsutil -q stat gs://main-bucket/sub-directory-bucket/*; echo $?

Result 0 means exists; 1 means not exists

like image 21
Kesavan Thiruvenkadasamy Avatar answered Nov 20 '22 09:11

Kesavan Thiruvenkadasamy


If your script allows for non-zero exit codes, then:

#!/bin/bash

file_path=gs://main-bucket/sub-directory-bucket/object1.gz
gsutil -q stat $file_path
status=$?

if [[ $status == 0 ]]; then
  echo "File exists"
else
  echo "File does not exist"
fi

But if your script is set to fail on error, then you can't use exit codes. Here is an alternative solution:

#!/bin/bash
trap 'exit' ERR

file_path=gs://main-bucket/sub-directory-bucket/object1.gz
result=$(gsutil -q stat $file_path || echo 1)
if [[ $result != 1 ]]; then
  echo "File exists"
else
  echo "File does not exist"
fi

like image 9
Igor-S Avatar answered Nov 20 '22 10:11

Igor-S


There is also gsutil ls (https://cloud.google.com/storage/docs/gsutil/commands/ls)

e.g.

gsutil ls gs://my-bucket/foo.txt

Output is either that same filepath or "CommandException: One or more URLs matched no objects."

like image 3
starmandeluxe Avatar answered Nov 20 '22 09:11

starmandeluxe


Simply using the ls command and counting the number of rows of the output.

If 0 then file not there, if 1 the file exists.

file_exists=$(gsutil ls gs://my_bucket/object1.gz | wc -l)

The same could be used for many files of course.

files_number=$(gsutil ls gs://my_bucket/object* | wc -l)
like image 2
Tycho Avatar answered Nov 20 '22 09:11

Tycho