AKS. Can't pull image from an acr

Question

I try to pull image from an ACR using a secret and I can't do it.

I created resources using azure cli commands:

az login
az provider register -n Microsoft.Network
az provider register -n Microsoft.Storage
az provider register -n Microsoft.Compute
az provider register -n Microsoft.ContainerService

az group create --name aksGroup --location westeurope

az aks create --resource-group aksGroup --name aksCluster --node-count 1 --generate-ssh-keys -k 1.9.2
az aks get-credentials --resource-group aksGroup --name aksCluster

az acr create --resource-group aksGroup --name aksClusterRegistry --sku Basic --admin-enabled true

After that I logged in and pushed image successfully to created ACR from local machine.

docker login aksclusterregistry.azurecr.io
docker tag jetty aksclusterregistry.azurecr.io/jetty
docker push aksclusterregistry.azurecr.io/jetty

The next step was creating a secret:

kubectl create secret docker-registry secret --docker-server=aksclusterregistry.azurecr.io --docker-username=aksClusterRegistry --docker-password=<Password from tab ACR/Access Keys> [email protected]

And eventually I tried to create pod with image from the ACR:

#pod.yml
apiVersion: v1
kind: Pod
metadata:
  name: jetty
spec:
  containers:
  - name: jetty
    image: aksclusterregistry.azurecr.io/jetty
  imagePullSecrets:
  - name: secret

kubectl create -f pod.yml

In result I have a pod with status ImagePullBackOff:

>kubectl get pods
NAME                    READY     STATUS             RESTARTS   AGE
jetty                   0/1       ImagePullBackOff   0          1m
> kubectl describe pod jetty
Events:
  Type     Reason                 Age              From                               Message
  ----     ------                 ----             ----                               -------
  Normal   Scheduled              2m               default-scheduler                  Successfully assigned jetty to aks-nodepool1-62963605-0
  Normal   SuccessfulMountVolume  2m               kubelet, aks-nodepool1-62963605-0  MountVolume.SetUp succeeded for volume "default-token-w8png"
  Normal   Pulling                2m (x2 over 2m)  kubelet, aks-nodepool1-62963605-0  pulling image "aksclusterregistry.azurecr.io/jetty"
  Warning  Failed                 2m (x2 over 2m)  kubelet, aks-nodepool1-62963605-0  Failed to pull image "aksclusterregistry.azurecr.io/jetty": rpc error: code = Unknown desc = Error response from daemon: Get https://aksclusterregistry.azurecr.io/v2/jetty/manifests/latest: unauthorized: authentication required
  Warning  Failed                 2m (x2 over 2m)  kubelet, aks-nodepool1-62963605-0  Error: ErrImagePull
  Normal   BackOff                2m (x5 over 2m)  kubelet, aks-nodepool1-62963605-0  Back-off pulling image "aksclusterregistry.azurecr.io/jetty"
  Normal   SandboxChanged         2m (x7 over 2m)  kubelet, aks-nodepool1-62963605-0  Pod sandbox changed, it will be killed and re-created.
  Warning  Failed                 2m (x6 over 2m)  kubelet, aks-nodepool1-62963605-0  Error: ImagePullBackOff

What's wrong? Why does approach with secret not work? Please don't advice me approach with service principal, because I would like to understand why this aproach doesn't work. I think it must be working.

Answer 1

The "old" way with AKS was to do create secret as you mentioned. That is no longer recommended.

The "new" way is to attach the container registry. This article explains the "new" way to attach ACR, and also provides a link to the old way to clear up confusion. When you create your cluster, attach with:

az aks create -n myAKSCluster -g myResourceGroup --attach-acr $MYACR

Or if you've already created your cluster, update it with:

az aks update -n myAKSCluster -g myResourceGroup --attach-acr $MYACR

Notes:

• $MYACR is just the name of your registry without the .azurecr.io. Ex: MYACR=foobar not MYACR=foobar.azurecr.io.

• After you attach your ACR, it will take a few minutes for the ImagePullBackOff to transition to Running.

Answer 2

This looks good to me as well. That said, the recommendation is not to use the admin account, rather a service principle. With the SP you gain some granular control over access rights to the ACR instance (read, contributor, owner).

This doc includes two methods for authentication between AKS and ACR using service principles.

https://learn.microsoft.com/en-us/azure/container-registry/container-registry-auth-aks

Answer 3

It's not exactly the question case. But I was having similar issue with utilization of Attach ACR approach. My problem was with Upper case characters in the registry name. Below warning was being generated by az cli.

Uppercase characters are detected in the registry name. When using its server url in docker commands, to avoid authentication errors, use all lowercase

So ensure to use all lowercases in ACR urls on Docker commands.