Slow read performance on Mongo under Centos 6

Question

I successfully basic configure my replica set, but i noticed slow read performance. I have 2 mongo servers and 1 arbiter in my replica set. All are running on Centos 6.3 minimal setup (just database).

I also have single mongo instance on another Windows server 2012.

Before i build my replica set, my web server was on same OS as Mongo (that windows instance). Performance was fine, results was like

C# .NET 4.5 (IIS and Mongo on same VM) Read 2000 rows : ~250ms at AvgObj 2600 bytes

Now with my replica set i have very slow read performance. When my web server access replica set, i`m getting result like

C# .NET 4.5 (separated, IIS and Mongo on Centos Replica) Read 2000 rows : ~2500ms at AvgObj 2600 bytes

I tried to read from my single Mongo instant from remote IIS and result is like

C# .NET 4.5 (separated, IIS and Single Mongo on Windows) Read 2000 rows : ~600ms at AvgObj 2600 bytes

All servers are running in single physical machine with Hyper-V core server.

Im using latest C# MongoDB Driver 1.7 version. From what i tested in command shell, on all mongo servers my query is executed in 10ms.

Can anyone give me suggestions why my replica set on separated Centos servers have slow read performance? is it network, drivers or what else?

My C# config is

            var server1 = new MongoServerAddress("1.0.0.1"); //primary
            var server2 = new MongoServerAddress("1.0.0.2"); /secundary
            var servers = new List<MongoServerAddress> {server1, server2};

            var safe = new MongoClientSettings
                {

                    Servers = servers,
                    ReplicaSetName = "rs0",
                    ConnectionMode = ConnectionMode.ReplicaSet,
                    WriteConcern = new WriteConcern
                        {
                            Journal = false,
                            W = 1
                        },
                    ReadPreference = new ReadPreference
                        {
                            ReadPreferenceMode = ReadPreferenceMode.PrimaryPreferred
                        }
                };

Answer 1

For best performance on Hyper V, make sure you are using the regular Network Adaptors (non-legacy), and that you install Integration Services inside of the guest operating system.

For CentOS 6.x, you can install the Integration Services from here: http://www.microsoft.com/en-us/download/details.aspx?id=34603

If you find that your VM was using a legacy network adapter, note that when you add a new adapter you'll have to reconfigure the networking inside of the guest (CentOS) operating system.

Answer 2

In the Mongo shell you are typically getting back the first 10 records - in .NET you are getting back the first 101 records or 1 MB (whichever comes first). If you are opening a cursor and then timing the iterations, you will find that this can be extremely slow compared to the 10 record retrieval from the shell.

Try running a JS loop to retrieve all records in the shell and see how long that takes. Note you should run both the shell and the C# connection through your mongos connection if you have one up and running.