What Service Account Does GKE Use to Access GCR

Question

One of the Google Kubernetes Engine (GKE) clusters ($GKE_CLUSTER) within a Google Cloud Platform (GCP) project ($GCP_PROJECT) seems to be unable to pull Docker Images from Google Container Registry (GCR):

kubectl config current-context

#=>

$GKE_CLUSTER

and:

kubectl get pods --namespace=$NAMESPACE

#=>

NAME        READY   STATUS             RESTARTS   AGE 
. . .       . . .   . . .              . . .      . . .
$SOME_POD   0/1     ImagePullBackOff   0          1m
. . .       . . .   . . .              . . .      . . .

and:

kubectl get events \
--field-selector involvedObject.name=$SOME_POD \
--namespace=$NAMESPACE

#=>

LAST SEEN   TYPE     REASON    OBJECT          MESSAGE
. . .       . . .    . . .     . . .           . . .
1m          Normal   BackOff   pod/$SOME_POD   Back-off pulling image "$SOME_IMAGE"
. . .       . . .    . . .     . . .           . . .

Where:

• $SOME_POD is of the form: deployment-replicaSet-pod
• $SOME_IMAGE is of the form: us.gcr.io/$GCP_PROJECT/name:tag

Listing only service accounts for $GCP_PROJECT gives:

gcloud projects get-iam-policy $GCP_PROJECT \
--filter="serviceAccount" \
--flatten="bindings[].members" \
--format="value(bindings.members.split(':').slice(1:).flatten())"

#=>

[email protected]
[email protected] 
[email protected]  
[email protected]
service-XXXXXXXXXXXX@container-engine-robot.iam.gserviceaccount.com
[email protected]  
[email protected]    
[email protected]
[email protected]

Which of these service accounts is GKE using to access Docker Images hosted by GCR?

Answer 1

Each GKE node has an IAM Service Account associated with it. By default, nodes are given the Compute Engine default service account, which you can find by navigating to the IAM section of the Cloud Console.

If you are using the non default Compute Engine service account you probably will need to grant the service account roles/storage.objectViewer role in the project. Check this link for more information.

Answer 2

Google Kubernetes Engine (GKE) doesn’t access Google Container Registry (GCR) directly: one or more node pools associated with the GKE cluster push and pull Docker images through the Container Registry API, authenticated and authorized through service accounts.

If your GKE cluster has a single node pool:

gcloud container clusters describe $GKE_CLUSTER_NAME \
--format='value(nodeConfig.serviceAccount)' \
--zone=$GKE_CLUSTER_ZONE

#=>

default

If your GKE cluster has multiple node pools:

gcloud container clusters describe $GKE_CLUSTER_NAME \
--zone=$GKE_CLUSTER_ZONE \
--format='value[delimiter="\n"](nodePools[].config.serviceAccount)'

#=>

default
not-default@$GCP_PROJECT_NAME.iam.gserviceaccount.com
. . .

default here is the “Compute Engine default service account”: [email protected].

Note: default serv. acct. differ from serv. acct. that you are able to create: beginning with a number and ending in a domain other than: @$GCP_PROJECT_NAME.iam.gserviceaccount.com.

In order for a serv. acct. to be able to pull Docker images from GCR it needs to have at least one associated role with the following associated permissions:

• storage.objects.get
• storage.objects.list

You might encounter:

WARNING: You do not appear to have access to project [$GCP_PROJECT_NAME] or it does not exist.

if the serv. acct. does not have at least one associated role with the following associated permissions:

• resourcemanager.projects.get
• resourcemanager.projects.list

The curated role that contains all four of the above permissions is the roles/storage.objectViewer role.

If you ever want to store Docker images in another GCP project, you just need to grant the serv. acct. used by the node pool(s) within your GKE cluster the roles/storage.objectViewer role in the project your would like to store the Docker images (more on that here).

Note: you may wish to use GCP Artifact Registry (AR) going forward. More on transitioning from GCR to AR here.