How to add encryption-provider-config option to kube-apiserver?

Question

I am using kubernetes 1.15.7 version.

I am trying to follow the link https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#understanding-the-encryption-at-rest-configuration to enable 'encryption-provider-config' option on 'kube-apiserver'.

I edited file '/etc/kubernetes/manifests/kube-apiserver.yaml' and provided below option

- --encryption-provider-config=/home/rtonukun/secrets.yaml

But after that I am getting below error.

The connection to the server 171.69.225.87:6443 was refused - did you specify the right host or port?

with all kubectl commands like 'kubectl get no'.

Mainy, how do I do these below two steps?

3. Set the --encryption-provider-config flag on the kube-apiserver to point to the location of the config file.

4. Restart your API server.

Answer 1

I've reproduced exactly your scenario, and I'll try to explain how I fixed it

Reproducing the same scenario

1. Create the encrypt file on /home/koopakiller/secrets.yaml:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: r48bixfj02BvhhnVktmJJiuxmQZp6c0R60ZQBFE7558=
    - identity: {}

Edit the file /etc/kubernetes/manifests/kube-apiserver.yaml and set the --encryption-provider-config flag:

   - --encryption-provider-config=/home/koopakiller/encryption.yaml

Save the file and exit.

When I checked the pods status got the same error:

$ kubectl get pods -A
The connection to the server 10.128.0.62:6443 was refused - did you specify the right host or port?

Troubleshooting

Since kubectl is not working anymore, I tried to look directly the running containers using docker command, then I see kube-apiserver container was recently recreated:

$ docker ps 
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
54203ea95e39        k8s.gcr.io/pause:3.1   "/pause"                 1 minutes ago       Up 1 minutes                            k8s_POD_kube-apiserver-lab-1_kube-system_015d9709c9881516d6ecf861945f6a10_0
...

Kubernetes store the logs of created pods on /var/log/pods directory, I've checked the kube-apiserver log file and found a valuable information:

{"log":"Error: error opening encryption provider configuration file "/home/koopakiller/encryption.yaml": open /home/koopakiller/encryption.yaml: no such file or directory\n","stream":"stderr","time":"2020-01-22T13:28:46.772768108Z"}

Explanation

Taking a look at manifest file kube-apiserver.yaml is possible to see the command kube-apiserver, it runs into container, so they need to have the encryption.yaml file mounted into container.

If you check the volumeMounts in this file, you could see that only the paths below is mounted in container by default:

• /etc/ssl/certs
• /etc/ca-certificates
• /etc/kubernetes/pki
• /usr/local/share/ca-certificates
• /usr/share/ca-certificates

    ...
        volumeMounts:
        - mountPath: /etc/ssl/certs
          name: ca-certs
          readOnly: true
        - mountPath: /etc/ca-certificates
          name: etc-ca-certificates
          readOnly: true
        - mountPath: /etc/kubernetes/pki
          name: k8s-certs
          readOnly: true
        - mountPath: /usr/local/share/ca-certificates
          name: usr-local-share-ca-certificates
          readOnly: true
        - mountPath: /usr/share/ca-certificates
          name: usr-share-ca-certificates
          readOnly: true
    ...

Based on the facts above, we can assume that apiserver failed to start because /home/koopakiller/encryption.yaml doesn't actually mounted into container.

How to solve

I can see 2 ways to solve this issue:

1st - Copy the encryption file to /etc/kubernetes/pki (or any of the path above) and change the path in /etc/kubernetes/kube-apiserver.yaml:

   - --encryption-provider-config=/etc/kubernetes/encryption.yaml

Save the file and wait apiserver restart.

2nd - Create a new volumeMounts in the kube-apiserver.yaml manifest to mount a custom directory from node into container.

Let's create a new directory in /etc/kubernetes/secret (home folder isn't a good location to leave config files =)).

Edit /etc/kubernetes/manifests/kube-apiserver.yaml:

...
    - --encryption-provider-config=/etc/kubernetes/secret/encryption.yaml
...
  volumeMounts:
    - mountPath: /etc/kubernetes/secret
      name: secret
      readOnly: true
...
  volumes:
    - hostPath:
      path: /etc/kubernetes/secret
      type: DirectoryOrCreate
    name: secret
...

After save the file kubernetes will mount the node path /etc/kubernetes/secret into the same path into the apiserver container, wait start completely and try to list your node again.

Please let know if that helped!