ImagePullBackOff unauthorized: authentication required

Question

I have gone through all the motions and I have what appears to be a common problem. Unfortunately, all of the solutions I've tried from github and SO have yet to work. Here's the error:

Warning  Failed     4m (x4 over 5m)    kubelet, aks-agentpool-97052351-0  Failed to pull image "ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi": [rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required, rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required, rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required]

-- created the service principal

az ad sp create-for-rbac 
  --scopes /subscriptions/11870e73-bdb2-47b0-bf27-25d24c41ae24/resourcegroups/USS-MicroService-Test/providers/Microsoft.ContainerRegistry/registries/UssMicroServiceRegistry
  --role Reader 
  --name kimage-reader

-- created the secret for Kube

kubectl create secret docker-registry kimagereadersecret --docker-server ussmicroserviceregistry.azurecr.io --docker-email [email protected] --docker-username=kimage-reader --docker-password 4b37b896-a04e-48b4-a950-5f1abdd3e7aa

-- kubectl.exe describe pod simpledotnetapi-deployment-6fbf97df55-2hg2m

Name:               simpledotnetapi-deployment-6fbf97df55-2hg2m
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               aks-agentpool-97052351-0/10.240.0.4
Start Time:         Mon, 17 Jun 2019 15:22:30 -0500
Labels:             app=simpledotnetapi-pod
                    pod-template-hash=6fbf97df55
Annotations:        <none>
Status:             Pending
IP:                 10.240.0.26
Controlled By:      ReplicaSet/simpledotnetapi-deployment-6fbf97df55
Containers:
  simpledotnetapi-simpledotnetapi:
    Container ID:
    Image:          ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi
    Image ID:
    Port:           5000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-hj9b5 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-hj9b5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-hj9b5
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                From                               Message
  ----     ------     ----               ----                               -------
  Normal   Scheduled  5m                 default-scheduler                  Successfully assigned default/simpledotnetapi-deployment-6fbf97df55-2hg2m to aks-agentpool-97052351-0
  Normal   BackOff    4m (x6 over 5m)    kubelet, aks-agentpool-97052351-0  Back-off pulling image "ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi"
  Normal   Pulling    4m (x4 over 5m)    kubelet, aks-agentpool-97052351-0  pulling image "ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi"
  Warning  Failed     4m (x4 over 5m)    kubelet, aks-agentpool-97052351-0  Failed to pull image "ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi": [rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required, rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required, rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required]
  Warning  Failed     4m (x4 over 5m)    kubelet, aks-agentpool-97052351-0  Error: ErrImagePull
  Warning  Failed     24s (x22 over 5m)  kubelet, aks-agentpool-97052351-0  Error: ImagePullBackOff

-- kubectl.exe get pod simpledotnetapi-deployment-6fbf97df55-2hg2m -o yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: 2019-06-17T20:22:30Z
  generateName: simpledotnetapi-deployment-6fbf97df55-
  labels:
    app: simpledotnetapi-pod
    pod-template-hash: 6fbf97df55
  name: simpledotnetapi-deployment-6fbf97df55-2hg2m
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: simpledotnetapi-deployment-6fbf97df55
    uid: a99e4ac8-8ec3-11e9-9bf8-86d46846735e
  resourceVersion: "813190"
  selfLink: /api/v1/namespaces/default/pods/simpledotnetapi-deployment-6fbf97df55-2hg2m
  uid: a1c220a2-913d-11e9-801a-c6aef815c06a
spec:
  containers:
  - image: ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi
    imagePullPolicy: Always
    name: simpledotnetapi-simpledotnetapi
    ports:
    - containerPort: 5000
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-hj9b5
      readOnly: true
  dnsPolicy: ClusterFirst
  imagePullSecrets:
  - name: kimagereadersecret
  nodeName: aks-agentpool-97052351-0
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-hj9b5
    secret:
      defaultMode: 420
      secretName: default-token-hj9b5
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2019-06-17T20:22:30Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2019-06-17T20:22:30Z
    message: 'containers with unready status: [simpledotnetapi_simpledotnetapi]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2019-06-17T20:22:30Z
    message: 'containers with unready status: [simpledotnetapi_simpledotnetapi]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: 2019-06-17T20:22:30Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi
    imageID: ""
    lastState: {}
    name: simpledotnetapi-simpledotnetapi
    ready: false
    restartCount: 0
    state:
      waiting:
        message: Back-off pulling image "ussmicroserviceregistry.azurecr.io/simpledotnetapi-simpledotnetapi"
        reason: ImagePullBackOff
  hostIP: 10.240.0.4
  phase: Pending
  podIP: 10.240.0.26
  qosClass: BestEffort
  startTime: 2019-06-17T20:22:30Z

-- yaml configuration file

apiVersion: apps/v1
kind: Deployment
metadata:
    name: simpledotnetapi-deployment
spec:
    replicas: 3
    selector:
        matchLabels:
            app: simpledotnetapi-pod
    template:
        metadata:
            labels:
                app: simpledotnetapi-pod
        spec:
            imagePullSecrets:
              - name: kimagereadersecret
            containers:
              - name: simpledotnetapi_simpledotnetapi
                image: ussmicroserviceregistry.azurecr.io/simpledotnetapi-simpledotnetapi
                ports:
                  - containerPort: 5000
---
apiVersion: v1
kind: Service
metadata:
    name: simpledotnetapi-service
spec:
    type: LoadBalancer
    ports:
    - port: 80
    selector:
       app: simpledotnetapi
       type: front-end

-- output of kubectl get secret kimagereadersecret

NAME                 TYPE                             DATA      AGE
kimagereadersecret   kubernetes.io/dockerconfigjson   1         1h

-- credentials/secret from Kube dashboard

{
  "kind": "Secret",
  "apiVersion": "v1",
  "metadata": {
    "name": "kimagereadersecret",
    "namespace": "default",
    "selfLink": "/api/v1/namespaces/default/secrets/kimagereadersecret",
    "uid": "86006aff-9156-11e9-801a-c6aef815c06a",
    "resourceVersion": "830006",
    "creationTimestamp": "2019-06-17T23:20:41Z"
  },
  "data": {
    ".dockerconfigjson": "eyJhdXRocyI6eyJ1c3NtaWNyb3NlcnZpY2VyZWdpc3RyeS5henVyZWNyLmlvIjp7InVzZXJuYW1lIjoiMzNjYjBjZTQtOTVmMC00NGJkLWJiYmYtNTZkNTA2ZmY0ZWIzIiwicGFzc3dvcmQiOiI0YjM3Yjg5Ni1hMDRlLTQ4YjQtYTk1MC01ZjFhYmRkM2U3YWEiLCJlbWFpbCI6ImNvcmV5cEB1bml0ZWQtc3lzdGVtcy5jb20iLCJhdXRoIjoiTXpOallqQmpaVFF0T1RWbU1DMDBOR0prTFdKaVltWXROVFprTlRBMlptWTBaV0l6T2pSaU16ZGlPRGsyTFdFd05HVXRORGhpTkMxaE9UVXdMVFZtTVdGaVpHUXpaVGRoWVE9PSJ9fX0="
  },
  "type": "kubernetes.io/dockerconfigjson"
}

-- Full dump from the Kube Dashboard

Failed to pull image "ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi": [rpc error: code = Unknown desc = Error response from daemon: manifest for ussmicroserviceregistry.azurecr.io/simpledotnetapi_simpledotnetapi:latest not found: manifest unknown: manifest unknown, rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required, rpc error: code = Unknown desc = Error response from daemon: Get https://ussmicroserviceregistry.azurecr.io/v2/simpledotnetapi_simpledotnetapi/manifests/latest: unauthorized: authentication required]

The entire project is in GitHub @ https://github.com/coreyperkins/KubeSimpleDotNetApi

-- ACR screenshot

-- Pod Failure in Kube

Answer 1

I'm fairly certain you didn't give it enough permissions:

az ad sp create-for-rbac 
  --scopes /subscriptions/11870e73-bdb2-47b0-bf27-25d24c41ae24/resourcegroups/USS-MicroService-Test/providers/Microsoft.ContainerRegistry/registries/UssMicroServiceRegistry
  --role Reader 
  --name kimage-reader

role should be acrpull, not reader. and just delete this secret: `kimagereadersecret 1 and reference to it in the pod. kubernetes will handle that for you.

Answer 2

Looks like you may be missing the kimagereadersecret in your Kubernetes cluster. As I understand az ad sp create-for-rbac just creates access to Azure resources, but how does k8s know which credentials to use to pull from the registry? You can follow this to create the registry secret. You can check that it exists with:

$ kubectl get secret kimagereadersecret

In your case, it could be that it's defaulting to no credentials or using whatever you have configured for Docker which doesn't have access to ussmicroserviceregistry.azurecr.io/simpledotnetapi-simpledotnetapi