Cloud ML Service account cannot access Cloud Storage and is not listed in IAM & admin panel

Question

When creating a new version of an ML Engine Model with the command

gcloud ml-engine versions create 'v1' --model=model_name --origin=gs://path_to_model/1/ --runtime-version=1.4

I recieve the following error:

ERROR: (gcloud.ml-engine.versions.create) FAILED_PRECONDITION: Field: version.deployment_uri Error: Read permissions are required for Cloud ML service account cloud-ml-service@**********.iam.gserviceaccount.com to the model file gs://path_to_model/1/saved_model.pb.
- '@type': type.googleapis.com/google.rpc.BadRequest
  fieldViolations:
  - description: Read permissions are required for Cloud ML service account cloud-ml-service@**********.iam.gserviceaccount.com to the model file gs://path_to_model/1/saved_model.pb.
  field: version.deployment_uri

This service account is not listed in the IAM & admin panel and does not belong to my project, so I don't want to grant permissions for this account manually.

Has anyone else also experienced this? Any suggestions on what I should do?

Additional information:

• The google storage bucket has storage class regional and location europe-west1.

• I already tried to disable (and re-enable) the ML Engine service with the command

  gcloud services disable ml.googleapis.com
  

  but this resulted in the following error:

  ERROR: (gcloud.services.disable) The operation with ID tmo-acf.********-****-****-****-************ resulted in a failure.
  

Updated information:

• The storage bucket does not belong to a different project.

• The command

  gcloud iam service-accounts get-iam-policy cloud-ml-service@**********.iam.gserviceaccount.com
  

  gives the error:

  ERROR: (gcloud.iam.service-accounts.get-iam-policy) PERMISSION_DENIED: Permission iam.serviceAccounts.getIamPolicy is required to perform this operation on service account projects/-/serviceAccounts/cloud-ml-service@**********.iam.gserviceaccount.com.
  

  The dash in the path projects/-/serviceAccounts/... in this error message seems very wrong to me.

PROBLEM HAS BEEN SOLVED

I was finally able to disable the ML Engine service after removing all my models. After re-enabling the service I got a new service account which shows up in my IAM & admin panel and is able to access my cloud storage.

Answer 1

If someone finds this issue, @freeCris wrote the solution in the question. I decided to write this down as I read all the documentation in the answers to find nothing useful and then realized he wrote how to solve it in the question itself.

For those wanting to fix this, just run (make sure you don't have resources in ML Engine such as models and versions):

gcloud services disable ml.googleapis.com

And then run:

gcloud services enable ml.googleapis.com

You'll get a new service account that this time is listed in your IAM console. Just add it to your GCS bucket and it'll work now.

Answer 2

I think the problem was, that you tried to create the model under a different project, which was not associated with that bucket you tried to reach. So you used the service account of that different project to access the bucket, that's why it did not have any permissions and did not appear in you AMI.

If that happens again or if anybody else has that problem, you can check your projects with gcloud projects list and change it with gcloud config set project <project name>.