Using Service Account with google cloud dataflow

Question

I am trying to run a Dataflow job with the service account. I am using the GcpOptions flags:

--serviceAccountKeyfile="dataflow-service-account.p12" --serviceAccountName="dataflow"

I am getting following error:

Unable to verify that GCS bucket exists.com.google.cloud.dataflow.sdk.util.DataflowPathValidator.verifyPathIsAccessible(DataflowPathValidator.java:84)
com.google.cloud.dataflow.sdk.util.DataflowPathValidator.validateOutputFilePrefixSupported(DataflowPathValidator.java:63)
com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.fromOptions(DataflowPipelineRunner.java:274)

I would like to use service account credentials against application default credentials.

Answer 1

I am posting an update on this question because the parameter name changed.

The parameter is now --serviceAccount in Java, and --service_account_email in Python.

The specified SA should have the following form: my-service-account-name@<project-id>.iam.gserviceaccount.com

Setting this value will change the worker service account, complete explanations could be found here.

Answer 2

serviceAccountName may be the full email address of the service account. That's usually how they are identified in google APIs. Usually something like: [email protected]

Also, you will need to add a whack of roles in IAM as per the docs.

I needed:

1. Compute Instance Admin
2. Dataflow Developer
3. Dataflow Worker
4. Logs Writer
5. Service Account Actor
6. Storage Object Admin