Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kubernetes pods can't pull images from container registry (gcp)

I want to update my deployment on kubernetes with a new image which exists on 'eu.gcr.io' (same project), I have done this before. But now the pods fail to pull the image because they are not authorized to do so. This is the error that we get in the pod logs.

Failed to pull image "eu.gcr.io/my-gcp-project/my-image:v1.009": 
rpc error: code = Unknown desc = Error response from daemon: 
unauthorized: You don't have the needed permissions to perform this operation,
and you may have invalid credentials.

The service account on the cluster has kubernetes admin and storage admin roles which should be sufficient. But even when I make the service account project editor (for debugging sake) it still doesn't work (same error).

I have also tried creating a fresh new cluster (default settings) and apply my deployment there, but then I got the exact same issue.

I'm not sure what I can try anymore.

Any help or suggestions are greatly appreciated.

EDIT:

I just found out that I can still pull and deploy older images. But every new image I build cannot be pulled by the kubernetes pods.

like image 472
Georges Lorré Avatar asked Mar 02 '19 10:03

Georges Lorré


People also ask

How to pull a private image from a Kubernetes cluster?

A Kubernetes cluster uses the Secret of kubernetes.io/dockerconfigjson type to authenticate with a container registry to pull a private image. If you already ran docker login, you can copy that credential into Kubernetes:

Why do pods fail to pull the new image on Kubernetes?

I want to update my deployment on kubernetes with a new image which exists on 'eu.gcr.io' (same project), I have done this before. But now the pods fail to pull the image because they are not authorized to do so. This is the error that we get in the pod logs.

What is imagepullsecrets in Kubernetes?

The imagePullSecrets field in the configuration file specifies that Kubernetes should get the credentials from a Secret named regcred. Learn more about Secrets. Learn more about using a private registry. Learn more about adding image pull secrets to a service account. See kubectl create secret docker-registry.

What is the status of the Kubernetes POD's status?

The Kubernetes pod's STATUS is ImagePullBackOff or ErrImagePull. To get detailed errors, run the following command and check Events from the output. We recommend that you start troubleshooting by checking the container registry's health and validating whether the container registry is accessible from the AKS cluster.


1 Answers

According to your desciption

I just found out that I can still pull and deploy older images. But every new image I build cannot be pulled by the kubernetes pods.

I assume you can pull docker image by command, but not kubectl.

docker pull eu.gcr.io/my-gcp-project/my-image:v1.009 

So reference by this article Using Google Container Registry with Kubernetes, the authenication is differnet between pull docker image by docker pull and kubectl .

Did you give access token to GKE?

kubectl create secret docker-registry gcr-access-token \
--docker-server=eu.gcr.io \
--docker-username=oauth2accesstoken \
--docker-password="$(gcloud auth print-access-token)" \
[email protected]
like image 83
howie Avatar answered Oct 03 '22 22:10

howie