Why a member of mongodb keep RECOVERING?

Question

I set up a replica set with three members and one of them is an arbiter.

One time I restart a member, the member keep RECOVERING for a long time and did not be SECONDARY again, even though the database was not large.

The status of replica set is like that:

rs:PRIMARY> rs.status()
{
        "set" : "rs",
        "date" : ISODate("2013-01-17T02:08:57Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 1,
                        "name" : "192.168.1.52:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 67968,
                        "optime" : Timestamp(1358388479000, 1),
                        "optimeDate" : ISODate("2013-01-17T02:07:59Z"),
                        "self" : true
                },
                {
                        "_id" : 2,
                        "name" : "192.168.1.50:29017",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 107,
                        "lastHeartbeat" : ISODate("2013-01-17T02:08:56Z"),
                        "pingMs" : 0
                },
                {
                        "_id" : 3,
                        "name" : "192.168.1.50:27017",
                        "health" : 1,
                        "state" : 3,
                        "stateStr" : "RECOVERING",
                        "uptime" : 58,
                        "optime" : Timestamp(1358246732000, 100),
                        "optimeDate" : ISODate("2013-01-15T10:45:32Z"),
                        "lastHeartbeat" : ISODate("2013-01-17T02:08:55Z"),
                        "pingMs" : 0,
                        "errmsg" : "still syncing, not yet to minValid optime 50f6472f:5d"
                }
        ],
        "ok" : 1
}

How should I fix this problem?

Answer 1

I had exact same issue: Secondary member of replica stuck in recovering mode. Here how to solve the issue:

1. stop secondary mongo db
2. delete all secondary db data files
3. start secondary mongo

It will start in startup2 mode and will replicate all data from Primary