Unable to bring up kubernetes API server [closed]

Question

Actual Issue:

Unable to start kubernetes API, due to which, unable to intite kube services like: kubectl version kubect get nodes

/home/ubuntu# kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}



   The connection to the server localhost:8080 was refused - did you specify the right host or port  ?

Background of the issue:

Docker is installed. Using below, kube components are installed:

apt-get update && apt-get install -y kubeadm kubelet kubectl

But, when executing kubeadm init --apiserver-advertise-address=$myip --ignore-preflight-errors=all:

I0408 09:09:07.316109       1 client.go:352] scheme "" not registered, fallback to default scheme
I0408 09:09:07.319904       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0  <nil>}]
I0408 09:09:07.323010       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
W0408 09:09:07.332669       1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
I0408 09:09:08.325625       1 client.go:352] parsed scheme: ""
I0408 09:09:08.325650       1 client.go:352] scheme "" not registered, fallback to default scheme
I0408 09:09:08.325707       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0  <nil>}]
I0408 09:09:08.325768       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 <nil>}]
W0408 09:09:08.326158       1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...

Getting the above in the kube api container logs. This is a fresh install .Also tried

sudo cp /etc/kubernetes/admin.conf $HOME/
sudo chown $(id -u):$(id -g) $HOME/admin.conf
export KUBECONFIG=$HOME/admin.conf

Expected Results:

kubectl version: should give only the version details without any connection issue message Kubectl get nodes: should give the details of Master node and status

Answer 1

I spent a couple of hours on that.
In my case, I was using Ubuntu 22.04 and Kubernetes 1.24.

API was restarting all the time, I didn't find something in kubelet logs:

service kubelet status 
journalctl -xeu kubelet

I checked the API logs through:

/var/log/containers/kube-apiserver-XXXX

I saw the same error:

grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379

Solution:

containerd config default | tee /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml  
service containerd restart
service kubelet restart  

Explanation:
The problem is cgroupv2 introduced with Ubuntu 21.04 and above(Debian since version 11).
You need to tell containerd to use the systemdCgroup driver via the config file(/etc/containerd/config.toml).
I'm using the default config file that containerd provides ( containerd config default | sudo tee /etc/containerd/config.toml).
Don't enable systemd_cgroup inside the "io.containerd.grpc.v1.cri" section, because the plugin doesn't seem support this anymore(status of the service will print following log:
"failed to load plugin io.containerd.grpc.v1.cri" error="invalid pluginconfig: systemd_cgroup only works for runtime io.containerd.runtime.v1.linux")
You need to enable the SystemdCgroup(=true) flag inside the section:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]"

Restart the containerd-service and then the kubelet-service or reboot your machine and then it should work as expected.

Credit: Dennis from slack