Recovering from Consul "No Cluster leader" state

Question

I have:

• one mesos-master in which I configured a consul server;
• one mesos-slave in which I configure consul client, and;
• one bootstrap server for consul.

When I hit start I am seeing the following error:

2016/04/21 19:31:31 [ERR] agent: failed to sync remote state: rpc error: No cluster leader 2016/04/21 19:31:44 [ERR] agent: coordinate update error: rpc error: No cluster leader

How do I recover from this state?

Answer 1

As of Consul 0.7 things work differently from Keyan P's answer. raft/peers.json (in the Consul data dir) has become a manual recovery mechanism. It doesn't exist unless you create it, and then when Consul starts it loads the file and deletes it from the filesystem so it won't be read on future starts. There are instructions in raft/peers.info. Note that if you delete raft/peers.info it won't read raft/peers.json but it will delete it anyway, and it will recreate raft/peers.info. The log will indicate when it's reading and deleting the file separately.

Assuming you've already tried the bootstrap or bootstrap_expect settings, that file might help. The Outage Recovery guide in Keyan P's answer is a helpful link. You create raft/peers.json in the data dir and start Consul, and the log should indicate that it's reading/deleting the file and then it should say something like "cluster leadership acquired". The file contents are:

[ { "id": "<node-id>", "address": "<node-ip>:8300", "non_voter": false } ]

where <node-id> can be found in the node-id file in the data dir.