ImagePullSecrets GCR

Question

I am having an issue configuring GCR with ImagePullSecrets in my deployment.yaml file. It cannot download the container due to permission

Failed to pull image "us.gcr.io/optimal-jigsaw-185903/syncope-deb": rpc error: code = Unknown desc = Error response from daemon: denied: Permission denied for "latest" from request "/v2/optimal-jigsaw-185903/syncope-deb/manifests/latest".

I am sure that I am doing something wrong but I followed this tutorial (and others like it) but with still no luck.

https://ryaneschinger.com/blog/using-google-container-registry-gcr-with-minikube/

The pod logs are equally useless:

"syncope-deb" in pod "syncope-deployment-64479cdcf5-cng57" is waiting to start: trying and failing to pull image

My deployment looks like:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  # Unique key of the Deployment instance
  name: syncope-deployment
  namespace: default
spec:
  # 3 Pods should exist at all times.
  replicas: 1
  # Keep record of 2 revisions for rollback
  revisionHistoryLimit: 2
  template:
    metadata:
      labels:
        # Apply this label to pods and default
        # the Deployment label selector to this value
        app: syncope-deb
    spec:
      imagePullSecrets:
      - name: mykey
      containers:
      - name: syncope-deb
        # Run this image
        image: us.gcr.io/optimal-jigsaw-185903/syncope-deb
        ports:
        - containerPort: 9080

Any I have a key in my default namespace called "mykey" that looks like (Edited out the Secure Data):

{"https://gcr.io":{"username":"_json_key","password":"{\n  \"type\": \"service_account\",\n  \"project_id\": \"optimal-jigsaw-185903\",\n  \"private_key_id\": \"EDITED_TO_PROTECT_THE_INNOCENT\",\n  \"private_key\": \"-----BEGIN PRIVATE KEY-----\\EDITED_TO_PROTECT_THE_INNOCENT\\n-----END PRIVATE KEY-----\\n\",\n  \"client_email\": \"[email protected]\",\n  \"client_id\": \"109145305665697734423\",\n  \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\n  \"token_uri\": \"https://accounts.google.com/o/oauth2/token\",\n  \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\n  \"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/bobs-service%40optimal-jigsaw-185903.iam.gserviceaccount.com\"\n}","email":"[email protected]","auth":"EDITED_TO_PROTECT_THE_INNOCENT"}}

I even loaded that user up with the permissions of:

• Editor Cloud Container
• Builder Cloud Container
• Builder Editor Service
• Account Actor Service
• Account Admin Storage
• Admin Storage Object
• Admin Storage Object Creator
• Storage Object Viewer

Any help would be appreciated as I am spending a lot of time on seemingly a very simple problem.

Answer 1

The issue is most likely caused by you using a secret of type dockerconfigjson and having valid dockercfg in it. The kubectl command changed at some point that causes this.

Can you check what it is marked as dockercfg or dockerconfigjson and then check if its valid dockerconfigjson.

The json you have provided is dockercfg (not the new format)

See https://github.com/kubernetes/kubernetes/issues/12626#issue-100691532 for info about the formats