Mongodb Replication Set Server Status: What does FATAL status actually mean?

Question

I have a mongo db replication set with three servers. The 3rd node shows that it is in a FATAL status:

{
        "_id" : 3,
        "name" : "slave2-1:27017",
        "health" : 1,
        "state" : 4,
        "stateStr" : "FATAL",
        "uptime" : 276,
        "optime" : {
            "t" : 1321929689000,
            "i" : 1
        },
        "optimeDate" : ISODate("2011-11-22T02:41:29Z"),
        "lastHeartbeat" : ISODate("2011-11-22T02:46:05Z"),
        "pingMs" : 0
}

When I log into the server in question, the mongo shell prompt shows that it is in fatal status. What does the FATAL status actually mean and how can I fix this?

Answer 1

Fatal state means that the member is in some state that it cannot recover from. You will need to intervene to get it back online. You should shut it down and diagnose the issue (if you don't know why it's in fatal state, there should be information about it in its log).

Common causes of fatal state are an invalid config, removing a node from the set (prior to 2.0), a failed rollback, or any other unhandled exceptional situation.