mongodb java driver - com.mongodb.MongoException: can't find a master

Question

I am trying to connect to a remote mongodb. I developed my application with the local mongodb. Now I deployed the application to the dev and configured the dev mongodb. I am getting the following exception.

Caused by: com.mongodb.MongoException: can't find a master
    at com.mongodb.DBTCPConnector.checkMaster(DBTCPConnector.java:509)
    at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:266)
    at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:289)
    at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:274)
    at com.mongodb.DBCursor._check(DBCursor.java:368)
    at com.mongodb.DBCursor._hasNext(DBCursor.java:459)
    at com.mongodb.DBCursor.hasNext(DBCursor.java:484)

The funny thing is that I can connect to the dev mongodb with the replica set server addresses from my local application, but when I try to have the application(deplyed into the dev) connect to the dev mongodb, I see the error above.

I am wondering if there is anybody who has faced the same issue and resolved it.

Answer 1

This confounding aspect of mongodb has it clashing with principles of political science in its voting policy.

Here is how it happens.

• A replica set exists; it must have an odd number of voting nodes.
• The primary node fails because the server / network goes down or is brought down. Other nodes may also fail but most importantly...
• An even number of nodes remain with no primary.
• The remaining even number of nodes cannot settle on a primary and are caught in a political deadlock (aka a hung parliament with no majority).
  • A re-election occurs but the primary is still down; its another deadlock. Loop here.

One solution is to sway the election by assigning weight to votes such that candidates are no longer equal. In the mongo world, this is done by assigning priority to members.

Priority Comparisons The priority setting affects elections. Members will prefer to vote for members with the highest priority value.

One does this by entering the mongo shell (on admin) and updating the rs.conf

cfg = rs.conf()
cfg.members[0].priority = 100
cfg.members[1].priority = 99
cfg.members[3].priority = 98
rs.reconfig(cfg)

Under this configuration, when primary member 0 fails then member 1 will be voted as primary.

Here are some good links:

http://docs.mongodb.org/manual/core/replica-set-elections/

http://docs.mongodb.org/manual/core/replica-set-architecture-four-members/

Finally, This situation is very common on a cloud architecture with techniques like availability sets, that is scaling up and down -- by time,cpu,load or by other metrics -- and should really be handled by a random or some unfair discriminating policy for all default replica sets. Even without techniques a primary on a default replica set is going to deadlock at some point, making it unavailable. A significant failure in my opinion.