Kubernetes pods can't pull images from container registry (gcp)

Question

I want to update my deployment on kubernetes with a new image which exists on 'eu.gcr.io' (same project), I have done this before. But now the pods fail to pull the image because they are not authorized to do so. This is the error that we get in the pod logs.

Failed to pull image "eu.gcr.io/my-gcp-project/my-image:v1.009": 
rpc error: code = Unknown desc = Error response from daemon: 
unauthorized: You don't have the needed permissions to perform this operation,
and you may have invalid credentials.

The service account on the cluster has kubernetes admin and storage admin roles which should be sufficient. But even when I make the service account project editor (for debugging sake) it still doesn't work (same error).

I have also tried creating a fresh new cluster (default settings) and apply my deployment there, but then I got the exact same issue.

I'm not sure what I can try anymore.

Any help or suggestions are greatly appreciated.

EDIT:

I just found out that I can still pull and deploy older images. But every new image I build cannot be pulled by the kubernetes pods.

Answer 1

According to your desciption

I just found out that I can still pull and deploy older images. But every new image I build cannot be pulled by the kubernetes pods.

I assume you can pull docker image by command, but not kubectl.

docker pull eu.gcr.io/my-gcp-project/my-image:v1.009 

So reference by this article Using Google Container Registry with Kubernetes, the authenication is differnet between pull docker image by docker pull and kubectl .

Did you give access token to GKE?

kubectl create secret docker-registry gcr-access-token \
--docker-server=eu.gcr.io \
--docker-username=oauth2accesstoken \
--docker-password="$(gcloud auth print-access-token)" \
[email protected]