How would I loop through all the file names in a subdirectory on Google Cloud Storage with python?

Question

Say I have some bucket/subdirectory on Google Cloud Storage and this bucket's address is:

gs://test-monkeys-example/training_data/cats

In this cats subdirectory I have a bunch of images of cats, all of which are jpgs. How would I in python loop through the cats subdirectory and print out all the names of the files in it?

Something like:

for x in directory('gs://test-monkeys-example/training_data/cats'):
    print(x)

Obviously directory('gs://test-monkeys-example/training_data/cats') is not how to do this and is just psuedocode- how would i do this?!

Answer 1

Google Cloud Storage supports listing only objects that begin with a certain prefix. You can access it from the client library like so:

from google.cloud import storage

client = storage.Client()
bucket = client.bucket('mybucket')
for blob in bucket.list_blobs(prefix='training_data/cats'):
  print blob.name

Answer 2

Use the storage module:

import google.datalab.storage as storage
cats = [o.key for o in storage.Bucket('test-monkeys-example').objects()
  if o.key.startswith('training_data/cats')]

This gives you a list of such cats.

Alternatively, you could use the Objects class:

cats = [o.key for o in storage.Objects('test-monkeys-example', '', '')
  if o.key.startswith('training_data/cats')]

If you don't need the list put in a variable, you can use the %gcs magic, it's easier:

%gcs list -o gs://test-monkeys-example/training_data/cats/*

This prints an HTML table of the keys. Note this is a full GCS path, starting with gs://.