Kubernetes imagePullSecrets not working; getting "image not found"

Question

I have an off-the-shelf Kubernetes cluster running on AWS, installed with the kube-up script. I would like to run some containers that are in a private Docker Hub repository. But I keep getting a "not found" error:

 > kubectl get pod NAME                      READY     STATUS                                        RESTARTS   AGE maestro-kubetest-d37hr    0/1       Error: image csats/maestro:latest not found   0          22m 

I've created a secret containing a .dockercfg file. I've confirmed it works by running the script posted here:

 > kubectl get secrets docker-hub-csatsinternal -o yaml | grep dockercfg: | cut -f 2 -d : | base64 -D > ~/.dockercfg  > docker pull csats/maestro latest: Pulling from csats/maestro 

I've confirmed I'm not using the new format of .dockercfg script, mine looks like this:

> cat ~/.dockercfg {"https://index.docker.io/v1/":{"auth":"REDACTED BASE64 STRING HERE","email":"[email protected]"}} 

I've tried running the Base64 encode on Debian instead of OS X, no luck there. (It produces the same string, as might be expected.)

Here's the YAML for my Replication Controller:

--- kind: "ReplicationController" apiVersion: "v1" metadata:   name: "maestro-kubetest" spec:   replicas: 1   selector:     app: "maestro"     ecosystem: "kubetest"     version: "1"   template:     metadata:       labels:         app: "maestro"         ecosystem: "kubetest"         version: "1"     spec:       imagePullSecrets:         - name: "docker-hub-csatsinternal"       containers:         - name: "maestro"           image: "csats/maestro"           imagePullPolicy: "Always"        restartPolicy: "Always"       dnsPolicy: "ClusterFirst" 

kubectl version:

Client Version: version.Info{Major:"1", Minor:"0", GitVersion:"v1.0.3", GitCommit:"61c6ac5f350253a4dc002aee97b7db7ff01ee4ca", GitTreeState:"clean"} Server Version: version.Info{Major:"1", Minor:"0", GitVersion:"v1.0.3", GitCommit:"61c6ac5f350253a4dc002aee97b7db7ff01ee4ca", GitTreeState:"clean"} 

Any ideas?

Answer 1

Another possible reason why you might see "image not found" is if the namespace of your secret doesn't match the namespace of the container.

For example, if your Deployment yaml looks like

apiVersion: extensions/v1beta1 kind: Deployment metadata:   name: mydeployment   namespace: kube-system 

Then you must make sure the Secret yaml uses a matching namespace:

apiVersion: v1 kind: Secret metadata:   name: mysecret   namespace: kube-system data:   .dockerconfigjson: **** type: kubernetes.io/dockerconfigjson 

If you don't specify a namespace for your secret, it will end up in the default namespace and won't get used. There is no warning message. I just spent hours on this issue so I thought I'd share it here in the hope I can save somebody else the time.

Answer 2

Docker generates a config.json file in ~/.docker/ It looks like:

{     "auths": {         "index.docker.io/v1/": {             "auth": "ZmFrZXBhc3N3b3JkMTIK",             "email": "[email protected]"         }     } } 

what you actually want is:

{"https://index.docker.io/v1/": {"auth": "XXXXXXXXXXXXXX", "email": "[email protected]"}} 

note 3 things:

• 1) there is no auths wrapping
• 2) there is https:// in front of the URL
• 3) it's one line

then you base64 encode that and use as data for the .dockercfg name

apiVersion: v1 kind: Secret metadata:    name: registry data:   .dockercfg: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX== type: kubernetes.io/dockercfg 

Note again the .dockercfg line is one line (base64 tends to generate a multi-line string)