How can I restore etcd cluster from snapshot in docker image on CoreOS?

Question

I have a Kubernetes cluster (v1.5.6) with 3 nodes etcd cluster (etcd version 3.1.5) on vmware. This etcd nodes are running in three docker containers(on three hosts) on coreos on vmware.

I try to backup etcd with the following solution:

docker run --rm --net=host -v /tmp:/etcd_backup -e ETCDCTL_API=3 quay.io/coreos/etcd:v3.1.5 etcdctl --endpoints=[1.1.1.1:2379,2.2.2.2:2379,3.3.3.3:2379] snapshot save etcd_backup/snapshot.db

The backup has been completed succesfully.

I want to create this kubernetes cluster from zero in another vmware environment, but I need to restore etcd from snapshot for this.

So far, I have not found the right solution that works with etcd in docker containers.

I try to restore with the following method, but unfortunately I did not succeed.

First, I created a new etcd node after I run the following command:

docker run --rm --net=host -v /tmp/etcd_bak:/etcd_backup -e ETCDCTL_API=3 registry:5000/quay.io/coreos/etcd:v3.1.5 etcdctl snapshot restore etcd_backup/snapshot.db --name etcd0 --initial-cluster etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380 --initial-cluster-token etcd-cluster-1 --initial-advertise-peer-urls http://etcd0:2380

Result:

2018-06-04 09:25:52.314747 I | etcdserver/membership: added member 7ff5c9c6942f82e [http://etcd0:2380] to cluster 5d1b637f4b7740d5
2018-06-04 09:25:52.314940 I | etcdserver/membership: added member 91b417e7701c2eeb [http://etcd2:2380] to cluster 5d1b637f4b7740d5
2018-06-04 09:25:52.315096 I | etcdserver/membership: added member faeb78734ee4a93d [http://etcd1:2380] to cluster 5d1b637f4b7740d5

Unfortunately, nothing happens.

What is the good solution to restore the etcd backup?

How do I create an empty etcd cluster/node and how should I restore the snapshot?

Answer 1

according to the Etcd Disaster Recovery document, you need restore all three etcd nodes from snapshot with commands like yours, then run three node with commands like this:

etcd \
  --name m1 \
  --listen-client-urls http://host1:2379 \
  --advertise-client-urls http://host1:2379 \
  --listen-peer-urls http://host1:2380 &

Also, you can extract etcdctl from the image, like this:

docker run --rm -v /opt/bin:/opt/bin registry:5000/quay.io/coreos/etcd:v3.1.5 cp /usr/local/bin/etcdctl /opt/bin

Then use etcdctl to restore snapshot:

# ETCDCTL_API=3 ./etcdctl snapshot restore snapshot.db \
  --name m1 \
  --initial-cluster m1=http://host1:2380,m2=http://host2:2380,m3=http://host3:2380 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-advertise-peer-urls http://host1:2380 \
  --data-dir /var/lib/etcd

This will restore snapshot to the /var/lib/etcd directory. Then start etcd with docker, don't forget mount /var/lib/etcd into your container, and specify --data-dir to it .