My kubernetes cluster IP address changed and now kubectl will no longer connect

Question

• Running under Ubuntu I used kubeadm init to setup my cluster (master node) and copied over the /etc/kubernetes/admin.conf $HOME/.kube/config and all was well when using kubectl.
• However after a reboot my master node has had an IP address change which is not the same as what is in $HOME/.kube/config so now I can no longer connect kubectl

So how do I regenerate the admin.conf now that I have a new IP address? Running kubeadm init will just kill everything which is not what I want.

Answer 1

I found this solution on the internet and it works for me:

    systemctl stop kubelet docker
    cd /etc/
    mv kubernetes kubernetes-backup
    mv /var/lib/kubelet /var/lib/kubelet-backup
    mkdir -p kubernetes
    cp -r kubernetes-backup/pki kubernetes
    rm kubernetes/pki/{apiserver.*,etcd/peer.*}
    systemctl start docker
    kubeadm init --ignore-preflight-errors=DirAvailable--var-lib-etcd
    #Run "kubeadm reset" on all nodes if was this error "error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
        [ERROR Port-10250]: Port 10250 is in use
        [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists"
    cp kubernetes/admin.conf ~/.kube/config
    kubectl get nodes --sort-by=.metadata.creationTimestamp
    kubectl delete node $(kubectl get nodes -o jsonpath='{.items[(@.status.conditions[0].status=="Unknown")].metadata.name}')
    kubectl get pods --all-namespaces

After These, Join your Slaves to Master. Reference: https://medium.com/@juniarto.samsudin/ip-address-changes-in-kubernetes-master-node-11527b867e88

Answer 2

You do not want to use kubeadm reset. That will reset everything and you would have to start configuring your cluster again.

Well, in your scenario, please have a look on the steps below:

1. nano /etc/hosts (update your new IP against YOUR_HOSTNAME)

2. nano /etc/kubernetes/config (configuration settings related to your cluster) here in this file look for the following params and update accordingly

   KUBE_MASTER="--master=http://YOUR_HOSTNAME:8080"

   KUBE_ETCD_SERVERS="--etcd-servers=http://YOUR_HOSTNAME:2379" #2379 is default port

3. nano /etc/etcd/etcd.conf (conf related to etcd)

   KUBE_ETCD_SERVERS="--etcd-servers=http://YOUR_HOSTNAME/WHERE_EVER_ETCD_HOSTED:2379"

   2379 is default port for etcd. and you can have multiple etcd servers defined here comma separated

4. Restart kubelet, apiserver, etcd services.

It is good to use hostname instead of IP to avoid such scenarios.

Hope it helps!