Permissions error with Apache Beam example on Google Dataflow

Question

I'm having trouble submitting an Apache Beam example from a local machine to our cloud platform.

Using gcloud auth list I can see that the correct account is currently active. I can use gsutil and the web client to interact with the file system. I can use the cloud shell to run pipelines through the python REPL.

But when I try and run the python wordcount example I get the following error:

IOError: Could not upload to GCS path gs://my_bucket/tmp: access denied.
Please verify that credentials are valid and that you have write access 
to the specified path.

Is there something I am missing with regards to the credentials?

Answer 1

Here are my two cents after spending the whole morning on the issue.

You should make sure that you login with gcloud on your local machine, however, pay attention to the warning message that return from gcloud auth login:

WARNING: `gcloud auth login` no longer writes application default credentials.

These credentials are required for the python code to identify your credentials properly.

Solution is rather simple, just use: gcloud auth application-default login

This will write a credentials file under: ~/.config/gcloud/application_default_credentials.json which is used for the authentication in the local development env.

Answer 2

You'll need to create a GCS bucket and folder for your project, then specify that as the pipeline parameter instead of using the default value.

https://cloud.google.com/storage/docs/creating-buckets