Connection to MongoDB replica set takes a minute+ in PHP when a secondary is unreachable

Question

I have a replica set consisting of 5 members: primary+arbiter on server 1, secondary+arbiter on server 2, hidden secondary (backup node) on server 3. I expect this configuration to work if one server goes down or loses connectivity temporarily.

However, when server 2 gone down (the one with secondary+arbiter nodes), I've encountered a weird problem. Any connection to the replica set from PHP was taking more than a minute. I tried modifying connection string to exclude the server that was down (secondary node) from it, but it did not help.

At the same time connecting via mongo console worked just fine. Primary node was remaining primary. PHP error log did not contain any errors.

The only thing that helped was removing the nodes on the server that gone down from the replica set.

However, I am now worried about the failover of the configuration. As I realize now, if the server with secondary+arbiter nodes will go down the whole configuration will stop working properly. Is there any way to avoid it? I need the PHP client to be able to connect to the primary regardless of whether secondary+arbiter server is available or not. How to achieve that?

Version of the mongo PHP client library is 1.6.x, version of the server is 3.0.

Answer 1

Your configuration will not work if server 2 goes down. Your replica set has 3 voting nodes, two of which are on server 2. If server 2 goes down, the set has 1/3 members up, and so cannot maintain or elect a primary. You need 2/3 able to talk to each other to elect a primary.

Your description of the problem is not consistent with how replica sets behave - connecting to the primary through the shell when 2/3 are down would not show the primary remaining primary. It would step down to secondary. Are you sure you have the right symptoms?

Answer 2

Just wondering what your connection string looks like.

In my experience something like this would work on a replica set configuration:

$m = new MongoClient("mongodb://rs1.example.com:27017,rs2.example.com:27017");

But when rs2 goes down, I would have similar experience as you described. You may want to make sure you use the correct connection string format:

$m = new MongoClient("mongodb://rs1.example.com:27017", array("replicaSet" => "myReplSetName"));

or

$m = new MongoClient("mongodb://rs1.example.com:27017,rs2.example.com:27017/?replicaSet=myReplSetName");