MongoDB Error: moveChunk failed to engage TO-shard in the data transfer: cannot start recv'ing chunk

Question

I am seeing below error in my mongodb changelog collection.

    {
    "_id" : "ip-xxx-xx-xx-xx-2016-05-04T06:31:55-5729975b16a41fe4f4618182",
    "server" : "ip-xxx-xx-xx-xx",
    "clientAddr" : "xxx.xx.xx.x:36339",
    "time" : ISODate("2016-05-04T06:31:55.754Z"),
    "what" : "moveChunk.from",
    "ns" : "my_db.my_collection",
    "details" : {
            "min" : {
                    "my_shard_key" : { "$minKey" : 1 }
            },
            "max" : {
                    "my_shard_key" : "H0285211-08"
            },
            "step 1 of 6" : 0,
            "step 2 of 6" : 278,
            "note" : "aborted",
            "errmsg" : "moveChunk failed to engage TO-shard in the data transfer: cannot start recv'ing chunk [{ my_shard_key: MinKey },{ my_shard_key: \"H0285211-08\" }) :: caused by :: could not query collection metadata :: caused by :: 8002 all servers down/unreachable when querying: my-conf-server-01:27019,my-conf-server-02:27019,my-conf-server-03:27019"
    }
    }

I can connect to my mongo config servers using mongo client. This confirms that there is no connectivity issue.

Let me know if more details are needed.

Answer 1

Found the problem.

This error occurs when there is no connectivity from primary node of replica set to config servers. In my case I successfully connected to my config servers(my-conf-server-01:27019,my-conf-server-02:27019,my-conf-server-03:27019) from one of the secondary nodes. Host entries were present there, but not on primary node.

Due to this problem, Mongo didn't start chunk migration and my shards were unbalanced.