Kubernetes Failed to pull image no basic auth credentials

Question

i'm trying to pull an image from github packages in kubernetes but i keep getting the error "no basic auth credentials"

i created a secret with this command:

kubectl create secret docker-registry regcred --docker-server=docker.pkg.github.com --docker-username=********* --docker-password=******* --docker-email=*****

and i added imagePullSecrets in the yaml file

i also have the config.json file with the credentials in $HOME/.docker/config.json in all the nodes of my cluster

Here is the content of the yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: invoice
  namespace: jhipster
spec:
  replicas: 1
  selector:
    matchLabels:
      app: invoice
      version: 'v1'
  template:
    metadata:
      labels:
        app: invoice
        version: 'v1'
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - invoice
                topologyKey: kubernetes.io/hostname
              weight: 100
      initContainers:
        - name: init-ds
          image: busybox:latest
          command:
            - '/bin/sh'
            - '-c'
            - |
              while true
              do
                rt=$(nc -z -w 1 invoice-mysql 3306)
                if [ $? -eq 0 ]; then
                  echo "DB is UP"
                  break
                fi
                echo "DB is not yet reachable;sleep for 10s before retry"
                sleep 10
              done
      containers:
        - name: invoice-app
          image: docker.pkg.github.com/jhipsterapps/kubernetes/invoice
          env:
            - name: SPRING_PROFILES_ACTIVE
              value: prod
            - name: SPRING_CLOUD_CONFIG_URI
              value: http://admin:${jhipster.registry.password}@jhipster-registry.jhipster.svc.cluster.local:8761/config
            - name: JHIPSTER_REGISTRY_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: registry-secret
                  key: registry-admin-password
            - name: EUREKA_CLIENT_SERVICE_URL_DEFAULTZONE
              value: http://admin:${jhipster.registry.password}@jhipster-registry.jhipster.svc.cluster.local:8761/eureka/
            - name: SPRING_DATASOURCE_URL
              value: jdbc:mysql://invoice-mysql.jhipster.svc.cluster.local:3306/invoice?useUnicode=true&characterEncoding=utf8&useSSL=false&useLegacyDatetimeCode=false&serverTimezone=UTC&createDatabaseIfNotExist=true
            - name: SPRING_SLEUTH_PROPAGATION_KEYS
              value: 'x-request-id,x-ot-span-context'
            - name: JAVA_OPTS
              value: ' -Xmx256m -Xms256m'
          resources:
            requests:
              memory: '512Mi'
              cpu: '500m'
            limits:
              memory: '1Gi'
              cpu: '1'
          ports:
            - name: http
              containerPort: 8081
          readinessProbe:
            httpGet:
              path: /management/health
              port: http
            initialDelaySeconds: 20
            periodSeconds: 15
            failureThreshold: 6
          livenessProbe:
            httpGet:
              path: /management/health
              port: http
            initialDelaySeconds: 120
      imagePullSecrets:
          - name: regcred

Here is the result of the describe command:

Name:         invoice-75859c6479-f9vmh
Namespace:    jhipster
Priority:     0
Node:         kworker1/10.66.12.213
Start Time:   Fri, 25 Sep 2020 16:35:50 +0200
Labels:       app=invoice
              pod-template-hash=75859c6479
              version=v1
Annotations:  <none>
Status:       Pending
IP:           10.244.1.117
IPs:
  IP:           10.244.1.117
Controlled By:  ReplicaSet/invoice-75859c6479
Init Containers:
  init-ds:
    Container ID:  docker://6d06e731b6fcdb4b8223ed0e0cd52687882413fe84fa18b17a853ce8cdf0ce65
    Image:         busybox:latest
    Image ID:      docker-pullable://busybox@sha256:d366a4665ab44f0648d7a00ae3fae139d55e32f9712c67accd604bb55df9d05a
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      while true
      do
        rt=$(nc -z -w 1 invoice-mysql 3306)
        if [ $? -eq 0 ]; then
          echo "DB is UP"
          break
        fi
        echo "DB is not yet reachable;sleep for 10s before retry"
        sleep 10
      done
      
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 25 Sep 2020 16:35:54 +0200
      Finished:     Fri, 25 Sep 2020 16:36:14 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-s8qbq (ro)
Containers:
  invoice-app:
    Container ID:   
    Image:          docker.pkg.github.com/jhipsterapps/kubernetes/invoice
    Image ID:       
    Port:           8081/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:      500m
      memory:   512Mi
    Liveness:   http-get http://:http/management/health delay=120s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:http/management/health delay=20s timeout=1s period=15s #success=1 #failure=6
    Environment:
      SPRING_PROFILES_ACTIVE:                 prod
      SPRING_CLOUD_CONFIG_URI:                http://admin:${jhipster.registry.password}@jhipster-registry.jhipster.svc.cluster.local:8761/config
      JHIPSTER_REGISTRY_PASSWORD:             <set to the key 'registry-admin-password' in secret 'registry-secret'>  Optional: false
      EUREKA_CLIENT_SERVICE_URL_DEFAULTZONE:  http://admin:${jhipster.registry.password}@jhipster-registry.jhipster.svc.cluster.local:8761/eureka/
      SPRING_DATASOURCE_URL:                  jdbc:mysql://invoice-mysql.jhipster.svc.cluster.local:3306/invoice?useUnicode=true&characterEncoding=utf8&useSSL=false&useLegacyDatetimeCode=false&serverTimezone=UTC&createDatabaseIfNotExist=true
      SPRING_SLEUTH_PROPAGATION_KEYS:         x-request-id,x-ot-span-context
      JAVA_OPTS:                               -Xmx256m -Xms256m
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-s8qbq (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-s8qbq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-s8qbq
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  5m57s                  default-scheduler  Successfully assigned jhipster/invoice-75859c6479-f9vmh to kworker1
  Normal   Pulling    5m56s                  kubelet, kworker1  Pulling image "busybox:latest"
  Normal   Pulled     5m54s                  kubelet, kworker1  Successfully pulled image "busybox:latest" in 1.873312412s
  Normal   Created    5m54s                  kubelet, kworker1  Created container init-ds
  Normal   Started    5m53s                  kubelet, kworker1  Started container init-ds
  Normal   BackOff    4m10s (x4 over 5m5s)   kubelet, kworker1  Back-off pulling image "docker.pkg.github.com/jhipsterapps/kubernetes/invoice"
  Normal   Pulling    3m58s (x4 over 5m32s)  kubelet, kworker1  Pulling image "docker.pkg.github.com/jhipsterapps/kubernetes/invoice"
  Warning  Failed     3m58s (x4 over 5m32s)  kubelet, kworker1  Failed to pull image "docker.pkg.github.com/jhipsterapps/kubernetes/invoice": rpc error: code = Unknown desc = Error response from daemon: Get https://docker.pkg.github.com/v2/jhipsterapps/kubernetes/invoice/manifests/latest: no basic auth credentials
  Warning  Failed     3m58s (x4 over 5m32s)  kubelet, kworker1  Error: ErrImagePull
  Warning  Failed     55s (x16 over 5m5s)    kubelet, kworker1  Error: ImagePullBackOff

Answer 1

The secret have to be in the same namespace as the deployment to be able to use it to pull from the docker registry.
So when you create the secret, you use:

kubectl create secret docker-registry regcred \
  --namespace=jhipster \ # <--
  --docker-server=docker.pkg.github.com \
  --docker-username=********* \
  --docker-password=******* \
  --docker-email=*****

It might also be an issue if you use 2fa and password (in the regcred), in which case you aught to create an access token to use as password instead.

Answer 2

For me, I had to add a https:// to the docker-server i.e.

kubectl create secret docker-registry aws-secret --docker-server=https://[email protected] --docker-username=AWS --docker-password=$PASSWORD -n NAMESPACE```