MongoDB replica set no primary, need to force new primary

Question

We have a mongoDB replica set which has 3 nodes;

1. Primary
2. Secondary
3. Arbiter

Somehow our replica set has ended up with 1 & 2 both being set as secondary members. I'm not sure how this has happened (we have had a server migration which one of the nodes runs on, but only one).

Anyway, I've been trying to re-elect a new primary for the replica set following the guide here

I'm unable to just use

rs.reconfig(cfg)

as it will only work if directed at the primary (which I don't have).

Using the force parameter

rs.reconfig(cfg, { force: true})

would appear to work but then when I requery the status of the replica set, both servers are still only showing as Secondary.

Why hasn't the force reconfig worked? Currently the database is locked out whatever I try.

Answer 1

1.Convert all nodes to standalone.

Stop mongod deamon and edit /etc/mongod.conf to comment replSet option.

Start mongod deamon.

2.Use mongodump to backup data for all nodes.

Reference from mongo docs:

https://docs.mongodb.com/manual/reference/program/mongodump/

3.Log into each node, and drop local database.

Doing this will delete replica set config on the node.

Or you can just delete a record in collection system.replset in local db, like it said here:

https://stackoverflow.com/a/31745150/4242454

4.Start all nodes with replSet option.

5.On the previous data node (not arbiter), initialize a new replica set.

6.Finally, reconfig replica set with rs.reconfig.