How does an odd number solve a split brain in a distributed system?

Question

Distributed system suggest using odd number of Master nodes, like 3 Master nodes or 5 Master nodes to avoid split brain problem.

But how does it solve the problem?

If there's 2 nodes ( A and B ), 1 Moderator, if A and B tell the Moderator that "I'm Master", then brain split occurs. The Moderator cannot decide which one is Master.

If there are 3 nodes ( A, B and C ), then if 2 nodes say Master is A, then the Moderator can decide that A is Master.

But what happens if A dies, then Moderator has to choose one Master from B and C, what if B and C tell Moderator that "I'm Master"? The brain split appears again.

Answer 1

Distributed systems don't typically suggest an odd number of nodes is what prevents split brain. Rather, it's majority quorums that avoid split brains. If a protocol chooses the node with a majority of votes to be the leader, and the protocol can ensure that nodes will only ever choose one leader, then logically there can only be one because there can only be one majority. Often, odd numbers of nodes are only used because they provide the greatest level of fault tolerance, e.g. a majority of 2 is 2, but a majority of 3 is also 2. That gives you room to tolerate one failure while still being able to acquire a majority of votes for a leader.

Of course, avoiding split brain isn't exactly just a matter of gathering votes from a majority of the nodes in a cluster and electing a leader from that. In asynchronous distributed systems, there's still a chance two nodes believe themselves to be the leader even when they're not, and all you need to create split brain is for two different clients to communicate with two different nodes that believe themselves to be the leader, whether they are or not. See Kyle Kingsbury's Jepsen blog for numerous examples of this.

Also, see the Raft paper for an example of a majority quorum based protocol that avoids split brain.