Why imagePullPolicy cannot be changed to other than Always in Kubernetes

Question

I have a kubernetes cluster set up and I would like to use local images. I have configured .yaml file so that it contains (in containers -> image -section) "imagePullPolicy: Never" like this:

spec:
      containers:
      - image: <name>:<version>
        name: <name>
        imagePullPolicy: Never
        resources: {}
      restartPolicy: Always

I have deployed this service to kubernetes but image cannot be pulled (getting ImagePullBackOff -error when viewing pods with kubectl get pod) since image cannot be found from internet/registry and for some unknown reason imagePullPolicy is in Always-value. This can be seen e.g. from /var/log/messages from text:

"spec":{"containers":[{"image":"<name>","imagePullPolicy":"Always","name":"<name>","

So my question is: Why is this imagePullPolicy in Always-value though I have set imagePullPolicy to Never in my .yaml file (which has of course been taken into use)? Is there some default value for imagePullPolicy that runs over the value described in .yaml file?

My environment is Centos7 and I'm using Kontena Pharos 2.2.0 (uses e.g. docker 1.13.1 (Apache License 2.0) and kubernetes 1.13.2 (Apache License 2.0)).

I expected that when I set "imagePullPolicy: Never" in .yaml file the value should now be Never (and not Always).

Thank you so much for helping!

Answer 1

welcome on StackOverflow.

It happens so, because your Kubernetes cluster has presumably enabled admission control plugin in api server, called 'AlwaysPullImages', which role is to overwrite (mutate) objects prior storing them in Kubernetes data store - etcd.

This is a default behavior of clusters bootstrapped with Kontena Pharos since version v2.0.0-alpha.2.

You can disable this admission plugin in your main cluster.yml config file:

...
addons:
  ingress-nginx:
    enabled: true

admission_plugins:
  - name: AlwaysPullImages
    enabled: false
... 

You should expect then to see PODs failing with different status reason, if image is not found on local registry:

client-deployment-99699599d-lfqmr    0/1     ErrImageNeverPull   0          42s

Please read more on using of Admission Controllers here