CoreOS - How to use a new token?

Question

We have the following problem. In our cluster the url's have changed. Once we changed the config to reflect those changes the urls did not get updated on 'discovery.etcd.io'. So our idea was to just use a new token. This however doesn't work. The cluster does not register with the new token on 'discovery.etcd.io'. We do not want to reinstall everytime we change the url or token. Is there a better way? Reinstalling works without a problem.

#cloud-config
hostname: server1
coreos:
  etcd2:
    # generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3
    discovery: https://discovery.etcd.io/<our token>
    # multi-region and multi-cloud deployments need to use $public_ipv4
    advertise-client-urls: server1:2379
    initial-advertise-peer-urls: server1:2380
    # listen on the official ports
    listen-client-urls: server1:2379
    listen-peer-urls: server1:2380
  #fleet:
  #    public-ip: server1
  #    metadata: region=eu-central-1
  #update:
  #  reboot-strategy: etcd-lock
  units:
    - name: etcd2.service
      command: start
   # - name: fleet.service
   #   command: start
ssh_authorized_keys:
   <our ssh keys>

Answer 1

You don't need to reinstall repeatedly. The following process is helpful for getting the cluster up step by step, rather than having a huge cloud-configure file which is difficult to debug.

1. Stop etcd and all dependent services (like flannel, fleet, etc that depend on etcd2): systemctl stop etcd2

2. Delete the etcd data files from /var/lib/etcd2/* (or the path in ETCD_DATA_DIR)

3. Change the discovery token in the cloud configure file stored in: /var/lib/coreos-install/user_data

4. Reboot.

Answer 2

discovery.etcd.io is only used for bootstrapping: you request a token for a number of hosts with https://discovery.etcd.io/new?size=3 for example and you basically reserve that URL to bootstrap your 3 hosts.

Once the cluster is bootstrapped, the nodes in the cluster now use their own local storage: your 3 nodes got to know each other through the discovery endpoint, and now form a cluster that has that information, so they don't need the discovery endpoint anymore.

So if you use a 'new' token, your cluster won't actually use it as it is already bootstrapped, because the nodes already form a cluster of their own. To bootstrap a new cluster, you need to delete the local data on each node.

I would suggest you read the migration documentation if you have other data you need to move to the new cluster.

the basic backup procedure is like:

etcdctl backup \
      --data-dir /var/lib/etcd \
      --backup-dir /tmp/etcd_backup