How to let kubelet communicate with apiserver by using HTTPS? v0.19

Question

I deployed apiserver on master node (core01) with following conf:

core01> /opt/bin/kube-apiserver \
  --insecure_bind_address=127.0.0.1 \
  --insecure_port=8080 \
  --kubelet_port=10250 \
  --etcd_servers=http://core01:2379,http://core02:2379,http://core03:2379 \
  --service-cluster-ip-range=10.1.0.0/16 \
  --allow_privileged=false \
  --logtostderr=true \
  --v=5 \
  --tls-cert-file="/var/run/kubernetes/apiserver_36kr.pem" \
  --tls-private-key-file="/var/run/kubernetes/apiserver_36kr.key" \
  --client-ca-file="/var/run/kubernetes/cacert.pem" \
  --kubelet-certificate-authority="/var/run/kubernetes/cacert.pem" \
  --kubelet-client-certificate="/var/run/kubernetes/kubelet_36kr.pem" \
  --kubelet-client-key="/var/run/kubernetes/kubelet_36kr.key"

On minion node (core02), I can call api from HTTPS:

core02> curl https://core01:6443/api/v1/nodes --cert /var/run/kubernetes/kubelet_36kr.pem --key /var/run/kubernetes/kubelet_36kr.key
> GET /api/v1/nodes HTTP/1.1
> Host: core01:6443
> User-Agent: curl/7.42.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Sat, 27 Jun 2015 15:33:50 GMT
< Content-Length: 1577
< 
{
  "kind": "NodeList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/nodes",
    "resourceVersion": "510078"
  }, ....

However, I can not start kubelet on this minion. It always complain no credentials.

How can I make it work? Is there any doc on master <-> minion communication authentication? Could you please give me the best practice?

---

FYI, The command is following:

core02> /opt/bin/kubelet \
  --logtostderr=true \
  --v=0 \
  --api_servers=https://core01:6443 \
  --address=127.0.0.1 \
  --port=10250 \
  --allow-privileged=false \
  --tls-cert-file="/var/run/kubernetes/kubelet_36kr.pem" \
  --tls-private-key-file="/var/run/kubernetes/kubelet_36kr.key"

kubelet log is following:

W0627 23:34:03.646311    3004 server.go:460] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Trying auth path instead.
W0627 23:34:03.646520    3004 server.go:422] Could not load kubernetes auth path /var/lib/kubelet/kubernetes_auth: stat /var/lib/kubelet/kubernetes_auth: no such file or directory. Continuing with defaults.
I0627 23:34:03.646710    3004 manager.go:127] cAdvisor running in container: "/system.slice/sshd.service"
I0627 23:34:03.647292    3004 fs.go:93] Filesystem partitions: map[/dev/sda9:{mountpoint:/ major:0 minor:30} /dev/sda4:{mountpoint:/usr major:8 minor:4} /dev/sda6:{mountpoint:/usr/share/oem major:8 minor:6}]
I0627 23:34:03.648234    3004 manager.go:156] Machine: {NumCores:1 CpuFrequency:2399996 MemoryCapacity:1046294528 MachineID:29f94a4fad8b31668bd219ca511bdeb0 SystemUUID:4F4AF929-8BAD-6631-8BD2-19CA511BDEB0 BootID:fa1bea28-675e-4989-ad86-00797721a794 Filesystems:[{Device:/dev/sda9 Capacity:18987593728} {Device:/dev/sda4 Capacity:1031946240} {Device:/dev/sda6 Capacity:113229824}] DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:21474836480 Scheduler:cfq} 8:16:{Name:sdb Major:8 Minor:16 Size:1073741824 Scheduler:cfq}] NetworkDevices:[{Name:eth0 MacAddress:52:54:71:f6:fc:b8 Speed:0 Mtu:1500} {Name:flannel0 MacAddress: Speed:10 Mtu:1472}] Topology:[{Id:0 Memory:1046294528 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}]}
I0627 23:34:03.649934    3004 manager.go:163] Version: {KernelVersion:4.0.5 ContainerOsVersion:CoreOS 695.2.0 DockerVersion:1.6.2 CadvisorVersion:0.15.1}
I0627 23:34:03.651758    3004 plugins.go:69] No cloud provider specified.
I0627 23:34:03.651855    3004 docker.go:289] Connecting to docker on unix:///var/run/docker.sock
I0627 23:34:03.652877    3004 server.go:659] Watching apiserver
E0627 23:34:03.748954    3004 reflector.go:136] Failed to list *api.Pod: the server has asked for the client to provide credentials (get pods)
E0627 23:34:03.750157    3004 reflector.go:136] Failed to list *api.Node: the server has asked for the client to provide credentials (get nodes)
E0627 23:34:03.751666    3004 reflector.go:136] Failed to list *api.Service: the server has asked for the client to provide credentials (get services)
I0627 23:34:03.758158    3004 plugins.go:56] Registering credential provider: .dockercfg
I0627 23:34:03.856215    3004 server.go:621] Started kubelet
E0627 23:34:03.858346    3004 kubelet.go:662] Image garbage collection failed: unable to find data for container /
I0627 23:34:03.869739    3004 kubelet.go:682] Running in container "/kubelet"
I0627 23:34:03.869755    3004 server.go:63] Starting to listen on 127.0.0.1:10250
E0627 23:34:03.899877    3004 event.go:185] Server rejected event '&api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba23275ceda25", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"starting", Message:"Starting kubelet.", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016043, nsec:856189989, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016043, nsec:856189989, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)
I0627 23:34:04.021297    3004 factory.go:226] System is using systemd
I0627 23:34:04.021790    3004 factory.go:234] Registering Docker factory
I0627 23:34:04.022241    3004 factory.go:89] Registering Raw factory
I0627 23:34:04.144065    3004 manager.go:946] Started watching for new ooms in manager
I0627 23:34:04.144655    3004 oomparser.go:183] oomparser using systemd
I0627 23:34:04.145379    3004 manager.go:243] Starting recovery of all containers
I0627 23:34:04.293020    3004 manager.go:248] Recovery completed
I0627 23:34:04.343829    3004 status_manager.go:56] Starting to sync pod status with apiserver
I0627 23:34:04.343928    3004 kubelet.go:1683] Starting kubelet main sync loop.
E0627 23:34:04.457765    3004 event.go:185] Server rejected event '&api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba232995c8213", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeReady", Message:"Node core02 status is now: NodeReady", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:452676115, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:452676115, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)
E0627 23:34:04.659874    3004 event.go:185] Server rejected event '&api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba232a599cf8c", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeReady", Message:"Node core02 status is now: NodeReady", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:658020236, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:658020236, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)

Answer 1

The first two lines of the kubelet log file actually point to the underlying problem -- you aren't specifying any client credentials for the kubelet to connect to the master.

The --tls-cert-file and --tls-private-key-file arguments for the kubelet are used to configure the http server on the kubelet (if not specified, the kubelet will generate a self-signed certificate for its https endpoint). This certificate / key pair are not used as the client certificate presented to the master for authentication.

To specify credentials, there are two options: a kubeconfig file and a kubernetes_auth file. The later is deprecated, so I would recommend using a kubeconfig file.

Inside the kubeconfig file you need to specify either a bearer token or a client certificate that the kubelet should present to the apiserver. You can also specify the CA certificate for the apiserver (if you want the connection to be secure) or tell the kubelet to skip checking the certificate presented by the apiserver. Since you have certificates for the apiserver, I'd recommend adding the CA certificate to the kubeconfig file.

The kubeconfig file should look like:

apiVersion: v1
kind: Config
users:
- name: kubelet
  user:
    client-certificate-data: <base64-encoded-cert>
    client-key-data: <base64-encoded-key>
clusters:
- name: local
  cluster:
    certificate-authority-data: <base64-encoded-ca-cert>
contexts:
- context:
    cluster: local
    user: kubelet
  name: service-account-context
current-context: service-account-context

To generate the base64 encoded client cert, you should be able to run something like cat /var/run/kubernetes/kubelet_36kr.pem | base64. If you don't have the CA certificate handy, you can replace the certificate-authority-data: <base64-encoded-ca-cert> with insecure-skip-tls-verify: true.

If you put this file at /var/lib/kubelet/kubeconfig it should get picked up automatically. Otherwise, you can use the --kubeconfig argument to specify a custom location.

Answer 2

All credit to jnoller as he specifies the below commands on this issue.
He just made a typo because he runs kubectl config set-credentials twice

This is similar to Robert Bailey's accepted answer except you don't need to base64 anything and it makes it easy to script.

kubectl config set-cluster default-cluster --server=https://${MASTER} \
    --certificate-authority=/path/to/ca.pem 

kubectl config set-credentials default-admin \
    --certificate-authority=/path/to/ca.pem \
    --client-key=/path/to/admin-key.pem \
    --client-certificate=/path/to/admin.pem      

kubectl config set-context default-system --cluster=default-cluster --user=default-admin
kubectl config use-context default-system

The resulting config generated in ~/.kube/config looks like this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /etc/kubernetes/certs/ca.crt
    server: https://kubernetesmaster
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    user: default-admin
  name: default-system
current-context: default-system
kind: Config
preferences: {}
users:
- name: default-admin
  user:
    client-certificate: /etc/kubernetes/certs/server.crt
    client-key: /etc/kubernetes/certs/server.key

Answer 3

As Robert Bailey said, the main problem has to do with the client credentials to connect the master ("Could not load kubeconfig file /var/lib/kubelet/kubeconfig..."). Instead of create the kubeconfig file manually, I chose to generate it using kubectl tool.

Example from docs:

$ kubectl config set-credentials myuser --username=myusername --password=mypassword
$ kubectl config set-cluster local-server --server=http://localhost:8080
$ kubectl config set-context default-context --cluster=local-server --user=myuser
$ kubectl config use-context default-context
$ kubectl config set contexts.default-context.namespace mynamespace

Those commands will generate a config file in ~/.kube/config

Check the result with:

$ kubectl config view

Then I just created a symbolic link inside /var/lib/kubelet (default place) to my config file:

$ cd /var/lib/kubelet
$ sudo ln -s ~/.kube/config kubeconfig

This worked for me. I hope it works for you too.