Mongodb replica set status showing "RECOVERING"

Question

I have setup the replica set over 3 mongo server and imported the 5 GB data. now status of secondary server showing "RECOVERING". Could you let me know what is means for "RECOVERING" and how to solve this issue.

Status is as below

rs.status()
{
    "set" : "kutendarep",
    "date" : ISODate("2013-01-15T05:04:18Z"),
    "myState" : 3,
    "members" : [
        {
            "_id" : 0,
            "name" : "10.1.4.138:27017",
            "health" : 1,
            "state" : 3,
            "stateStr" : "RECOVERING",
            "uptime" : 86295,
            "optime" : Timestamp(1357901076000, 4),
            "optimeDate" : ISODate("2013-01-11T10:44:36Z"),
            "errmsg" : "still syncing, not yet to minValid optime 50f04941:2",
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "10.1.4.21:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 86293,
            "optime" : Timestamp(1358160135000, 18058),
            "optimeDate" : ISODate("2013-01-14T10:42:15Z"),
            "lastHeartbeat" : ISODate("2013-01-15T05:04:18Z"),
            "pingMs" : 0
        },
        {
            "_id" : 2,
            "name" : "10.1.4.88:27017",
            "health" : 1,
            "state" : 3,
            "stateStr" : "RECOVERING",
            "uptime" : 86291,
            "optime" : Timestamp(1357900674000, 10),
            "optimeDate" : ISODate("2013-01-11T10:37:54Z"),
            "lastHeartbeat" : ISODate("2013-01-15T05:04:16Z"),
            "pingMs" : 0,
            "errmsg" : "still syncing, not yet to minValid optime 50f04941:2"
        }
    ],
    "ok" : 1

Answer 1

Login to RECOVERING instance.
Check RECOVERING instance replication status with,
db.printReplicationInfo()
You will get a result like this,

oplog first event time: Tue Jul 30 2019 17:26:37 GMT+0000 (UTC)
oplog last event time: Wed Jul 31 2019 16:46:53 GMT+0000
now: Thu Aug 22 2019 07:36:38 GMT+0000 (UTC)

If you find the difference between oplog last event time and now.

That means this particular instance is not PRIMARY and SECONDARY and not an active member of the replica set.

Now there are two solutions for this
First,
1. Login to RECOVERING instance
2. Delete data from existing db which will be /data/db
3. Restart this RECOVERING instance
4. (optional) If you find the following error. Remove that mongod.pid from the specified location.

Error starting mongod. /var/run/mongod/mongod.pid

5. Restart instance.
6. Now your recovering instance will be running state
and It will show PRIMARY or Secondary in place of RECOVERING.

Second,
Copy other running instance data into RECOVERING instance and restart mongodb.

Answer 2

The message on the "RECOVERING" replica set nodes means that these are still performing the initial sync. These nodes are not available for reads until they transitions to the Secondary state.

There are several steps in the initial sync.

See here for more information about the replica set synchronization process: http://docs.mongodb.org/manual/core/replica-set-sync/