Can you pass a keyfile.json to gsutil?

Question

I have a (maybe unique?) use case in some Python scripts that I am running. Namely, I want the parallel awesomeness of gsutil and so I don't do from google.cloud import storage, rather I use subprocess calls such as:

subprocess.Popen(["gsutil", "-q", "-m", "-o", "GSUtil:parallel_process_count=8,GSUtil:parallel_thread_count=8", "cp", files, destination])

in order to upload and download files from buckets.

In an instance group template I can pass in the service account via -scopes, but I'd like authentication to be handled at the application level. I tried setting environment variables and passing it to subprocess:

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "keyfile.json"
tmp_env = os.environ.copy()
subprocess.Popen(['gsutil', ...], env=tmp_env)

but to no avail. Running:

gcloud auth activate-service-account --key-file /path/to/keyfile.json --project my-project -q

seems to be the best way to authenticate with a json keyfile that does not require the Python API. But it doesn't work if I throw it in at the end of my Dockerfile, and while I could of course throw it in at the end of a startup.sh script that I have executed at the end of an instance group template embedded bootstrap.sh script, neither is really accomplishing what I'd like. Namely, both get away from my original goal of having "gsutil authentication" at the application level.

tl;dr Is there a way to pass keyfile.json credentials to gsutil? Is this a feature the gsutil team has ever discussed? My apologies if I just haven't been hunting the Cloud Platform and gsutil docs well enough.

Answer 1

You can provide a pointer to a JSON key file for gsutil in your .boto configuration file like so:

[Credentials]
gs_service_key_file = /path/to/your/keyfile.json

This is equivalent to running gsutil config -e for a standalone (non-gcloud) install.

If you want to provide this on the command line as opposed to in your .boto configuration file, you can use the -o parameter similar to how you configured the process and thread counts in your command line. To wit:

subprocess.Popen(["gsutil", "-q", "-m", "-o", "Credentials:gs_service_key_file=/path/to/your/keyfile.json",
                 "-o", "GSUtil:parallel_process_count=8", "-o", GSUtil:parallel_thread_count=8", "cp", files, destination])

Note that you need to make sure the key file path is accessible from within your container.