Failed to retrieve token from the Google Compute Engine metadata service. Status: 404

Question

I'm trying to set up Cloud SQL Proxy running as a sidecar in my GKE cluster. The configuration is done via Terraform. I've set up workload identity, required service accounts, and so on. When launching ./cloud_sql_proxy from within the GKE cluster (kubectl run -it --image google/cloud-sdk:slim --serviceaccount ksa-name --namespace k8s-namespace workload-identity-test), I get the following output:

root@workload-identity-test:/# ./cloud_sql_proxy -instances=project-id:europe-west4:db-instance=tcp:5432
2020/11/24 17:18:39 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
2020/11/24 17:18:40 GcloudConfig: error reading config: exit status 1; stderr was:
ERROR: (gcloud.config.config-helper) There was a problem refreshing your current auth tokens: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token from the Google Compute Enginemetadata service. Status: 404 Response:\nb'Unable to generate access token; IAM returned 404 Not Found: Requested entity was not found.\\n'", <google_auth_httplib2._Response object at 0x7fc5575545f8>)
Please run:

  $ gcloud auth login

to obtain new credentials.

If you have already logged in with a different account:

    $ gcloud config set account ACCOUNT

to select an already authenticated account to use.
2020/11/24 17:18:41 GcloudConfig: error reading config: exit status 1; stderr was:
ERROR: (gcloud.config.config-helper) There was a problem refreshing your current auth tokens: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token from the Google Compute Enginemetadata service. Status: 404 Response:\nb'Unable to generate access token; IAM returned 404 Not Found: Requested entity was not found.\\n'", <google_auth_httplib2._Response object at 0x7f06f72f45c0>)
Please run:

  $ gcloud auth login

to obtain new credentials.

If you have already logged in with a different account:

    $ gcloud config set account ACCOUNT

to select an already authenticated account to use.
2020/11/24 17:18:41 errors parsing config:
        Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/project-id/instances/europe-west4~db-instance?alt=json&prettyPrint=false": metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.admin" not defined

Here's troubleshooting I've done so far:

root@workload-identity-test:/# gcloud auth list
                   Credentialed Accounts
ACTIVE  ACCOUNT
*       [email protected]

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

λ gcloud container clusters describe mycluster --format="value(workloadIdentityConfig.workloadPool)"
project-id.svc.id.goog

λ gcloud container node-pools describe mycluster-node-pool --cluster=mycluster --format="value(config.workloadMetadataConfig.mode)"
GKE_METADATA

λ gcloud container node-pools describe mycluster-node-pool --cluster=mycluster--format="value(config.oauthScopes)"
https://www.googleapis.com/auth/monitoring;https://www.googleapis.com/auth/devstorage.read_only;https://www.googleapis.com/auth/logging.write;https://www.googleapis.com/auth/cloud-platform;https://www.googleapis.com/auth/userinfo.email;https://www.googleapis.com/auth/compute;https://www.googleapis.com/auth/sqlservice.admin

λ kubectl describe serviceaccount --namespace k8s-namespace ksa-name
Name:                ksa-name
Namespace:           k8s-namespace
Labels:              <none>
Annotations:         iam.gke.io/gcp-service-account: [email protected]
Image pull secrets:  <none>
Mountable secrets:   ksa-name-token-87n4t
Tokens:              ksa-name-token-87n4t
Events:              <none>

λ gcloud iam service-accounts get-iam-policy [email protected]
bindings:
- members:
  - serviceAccount:project-id.svc.id.goog[k8s-namespace/ksa-name]
  role: roles/iam.workloadIdentityUser
etag: BwW02zludbY=
version: 1

λ kubectl get networkpolicy --namespace k8s-namespace
No resources found in k8s-namespace namespace.

λ gcloud projects get-iam-policy project-id
bindings:
- members:
  - serviceAccount:[email protected]
  role: roles/cloudsql.editor

Expected result (I got this running on another cluster and changed configuration afterwards, can't find where my mistake is):

root@workload-identity-test:~# ./cloud_sql_proxy -instances=project-id:europe-west4:db-instance-2=tcp:5432
2020/11/24 18:09:54 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
2020/11/24 18:09:56 Listening on 127.0.0.1:5432 for project-id:europe-west4:db-instance-2
2020/11/24 18:09:56 Ready for new connections

What am I doing wrong? How do I troubleshoot or debug further?

Answer 1

This could be due to the service account not being enabled when the Kubernetes cluster was created, or it wasn't configured properly. Try checking if the Service Account is disabled and Enable if it is. You could also try to create a new service account and change the service account in the pods. Or finally, try to provide the credentials to the gcloud command when running.