Kubernetes does not start after restart system (Ubuntu)

Question

I installed K8s on my two Ubuntus in VirtualBox (Master and Node01). After installation (I proceeded according K8s doc site) I typed kubectl get nodes and got bot servers in status Ready. But after restart systems I got this:

# kubectl get nodes 
The connection to the server localhost:8080 was refused - did you specify the 
right host or port? 

I checked kubelet service and it is running:

# systemctl status kubelet
kubelet.service - kubelet: The Kubernetes Node Agent 
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) 
  Drop-In: /etc/systemd/system/kubelet.service.d 
           └─10-kubeadm.conf 
   Active: active (running) since Mon 2017-04-24 10:01:51 CEST; 15min ago 
     Docs: http://kubernetes.io/docs/ 
Main PID: 13128 (kubelet) 
    Tasks: 21 
   Memory: 48.2M 
      CPU: 58.014s 
   CGroup: /system.slice/kubelet.service 
           ├─13128 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --cluster-dns=10.96.0.10 --cluster-domain=cluster.local 
           └─13164 journalctl -k -f 

Apr 24 10:16:40 master kubelet[13128]: I0424 10:16:40.204156   13128 kuberuntime_manager.go:752] Back-off 5m0s restarting failed container=weave pod=weave-net-5qgvz_kube-system(4b7bb2f0-2691-11e7-bfb6-080027229776) 
Apr 24 10:16:40 master kubelet[13128]: E0424 10:16:40.204694   13128 pod_workers.go:182] Error syncing pod 4b7bb2f0-2691-11e7-bfb6-080027229776 ("weave-net-5qgvz_kube-system(4b7bb2f0-2691-11e7-bfb6-080027229776)"), skipping: fail 
Apr 24 10:16:42 master kubelet[13128]: I0424 10:16:42.972302   13128 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2b59d0d9-2692-11e7-bfb6-080027229776-default-token-h3v7c" (spec.Name: " 
Apr 24 10:16:48 master kubelet[13128]: I0424 10:16:48.949731   13128 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2bb42bc1-2692-11e7-bfb6-080027229776-default-token-h3v7c" (spec.Name: " 
Apr 24 10:16:51 master kubelet[13128]: I0424 10:16:51.978663   13128 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2b023c31-2692-11e7-bfb6-080027229776-default-token-h3v7c" (spec.Name: " 
Apr 24 10:16:52 master kubelet[13128]: I0424 10:16:52.909589   13128 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/4b7bb2f0-2691-11e7-bfb6-080027229776-default-token-gslqd" (spec.Name: " 
Apr 24 10:16:53 master kubelet[13128]: I0424 10:16:53.186057   13128 kuberuntime_manager.go:458] Container {Name:weave Image:weaveworks/weave-kube:1.9.4 Command:[/home/weave/launch.sh] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env: 
Apr 24 10:16:53 master kubelet[13128]: I0424 10:16:53.188091   13128 kuberuntime_manager.go:742] checking backoff for container "weave" in pod "weave-net-5qgvz_kube-system(4b7bb2f0-2691-11e7-bfb6-080027229776)" 
Apr 24 10:16:53 master kubelet[13128]: I0424 10:16:53.188717   13128 kuberuntime_manager.go:752] Back-off 5m0s restarting failed container=weave pod=weave-net-5qgvz_kube-system(4b7bb2f0-2691-11e7-bfb6-080027229776) 
Apr 24 10:16:53 master kubelet[13128]: E0424 10:16:53.189136   13128 pod_workers.go:182] Error syncing pod 4b7bb2f0-2691-11e7-bfb6-080027229776 ("weave-net-5qgvz_kube-system(4b7bb2f0-2691-11e7-bfb6-080027229776)"), skipping: fail 

Here is systemd log file with restarted kubelet: Google Drive.

... I'm not sure what I missed in doc or what happend with kubelet. Can I ask you for help? :]

• Ubuntu version

cat /etc/os-release 
NAME="Ubuntu" 
VERSION="16.04.2 LTS (Xenial Xerus)" 
ID=ubuntu 
ID_LIKE=debian 
PRETTY_NAME="Ubuntu 16.04.2 LTS" 
VERSION_ID="16.04" 
HOME_URL="http://www.ubuntu.com/" 
SUPPORT_URL="http://help.ubuntu.com/" 
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" 
VERSION_CODENAME=xenial 
UBUNTU_CODENAME=xenial 

• Kernel

# uname -a 
Linux ubuntu 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux 

• Kubectl version

# kubectl version 
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} 
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-03-28T16:24:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} 

• Kubeadm version

# kubeadm version 
kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} 

• Kubelet version

# kubelet --version 
Kubernetes v1.6.1 

• Docker version

# docker version 
Client: 
Version:      1.11.2 
API version:  1.23 
Go version:   go1.5.4 
Git commit:   b9f10c9 
Built:        Wed Jun  1 22:00:43 2016 
OS/Arch:      linux/amd64 

Server: 
Version:      1.11.2 
API version:  1.23 
Go version:   go1.5.4 
Git commit:   b9f10c9 
Built:        Wed Jun  1 22:00:43 2016 
OS/Arch:      linux/amd64 

Answer 1

I had a bad exported variable KUBECONFIG which is needed by kubelet (history details are in a comment under question).

To ~/.zprofile I saved KUBECONFIG=$HOME/admin.conf which solved my problem.

After reloading ENV variables is kubelet working:

# kubectl get nodes
NAME      STATUS     AGE       VERSION
master    Ready      5d        v1.6.1
node01    NotReady   5d        v1.6.1

Answer 2

I had the same problem with kubernetes 1.12.3 and ubuntu 16.04.05. I then looked at the kubernetes log by running the command

$ journalctl -u kubelet

and then in the log i saw that k8s was complaining (exiting with status 255) about swap being on.

So i then turned swap off by running

$ swapoff -a

Then i edited fstab and commented out the entry for swap

$ vi /etc/fstab
#comment out line with swap

and then rebooting the system. After the system came back up, i verified that swap was disabled by running

$ free -m

and checking whether the row for swap has 0. It did.

Then i verified that kubeapi service had successfully started by executing

$ systemctl status kubelet

It has successfully started. I verified by also re-checking journalctl logs. Did not see swap error this time.

I verified k8s node status by running

$ kubectl get nodes

which was now working and showing expected output.

NOTE: I had KUBECONFIG set in my .bash_profile file as well, previously.

root@k8s-master:~# cat .bash_profile
export KUBECONFIG="/etc/kubernetes/admin.conf"

Answer 3

As the comment there, you really need check whether apiserver is started, because kubectl will talk to apiserver. While from you description and version of kubeadm, I believe this is a duplicate question I just answered, so I just copy the answer to here.

---

In current version of kubeadm(v1.6.1), insecure port of ApiServer is abandoned by default, you can verify this by checking api-server yaml file in /etc/kubernetes/manifests/kube-apiserver.yaml, there is kube-apiserver parameter --insecure-port=0.

You can

• Correct this in a running cluster:

  $ mv kube-apiserver.yaml ../kube-apiserver.yaml
  // edit ../kube-apiserver.yaml to remove --insecure-port=0 
  // or change it to --insecure-port=<WHATERER_YOUR_LIKE>
  $ mv ../kube-apiserver.yaml kube-apiserver.yaml
  

• Do it right at startup. You need a kubeadm config file to do this. A simple one would like:

  apiVersion: kubeadm.k8s.io/v1alpha1
  kind: MasterConfiguration
  apiServerExtraArgs:
    insecure-port: 8080 //or whatever you like
  
  // Then you can start a master node use `kubeadm init --config=<this-configure-file-path>`