MongoDB C# 2.0 TimeoutException

Question

We've recently upgraded our web application to MongoDB C# Driver 2.0 and deployed to production. Below a certain load, the application runs fine. Once the load on the production server exceeds a certain limit, the CPU of the application instantly falls down to 0 and after about 30 seconds, this exception is logged several times:

System.TimeoutException message: A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = ReadPreferenceServerSelector{ ReadPreference = { Mode = Primary, TagSets = System.Collections.Generic.List`1[MongoDB.Driver.TagSet] } }, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", Type : "Standalone", State : "Disconnected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/10.4.0.113:27017" }", EndPoint: "Unspecified/10.4.0.113:27017", State: "Disconnected", Type: "Unknown" }] }.
stack trace:
at MongoDB.Driver.Core.Clusters.Cluster.ThrowTimeoutException(IServerSelector selector, ClusterDescription description)
at MongoDB.Driver.Core.Clusters.Cluster.<WaitForDescriptionChangedAsync>d__18.MoveNext()
--- End of stack trace

We are using a singleton MongoClient object, which is initiated like this:

private static object _syncRoot = new object();

private static MongoClient _client;
private static IMongoDatabase _database;

private IMongoDatabase GetDatabase()
{
    ...

    if (_client == null)
    {
        lock (_syncRoot)
        {
            if (_client == null)
            {
                _client = new MongoClient(
                    new MongoClientSettings
                    {
                        Server = new MongoServerAddress(host, port),
                        Credentials = new[] { credentials },
                    });

                _database = _client.GetDatabase("proddb");
                return _database;
            }
        }
    }
    return _database;
}

public IMongoCollection<T> GetCollection<T>(string name)
{
    return GetDatabase().GetCollection<T>(name);
}

A typical call to database looks like this:

public async Task<MongoItem> GetById(string id)
{
    var collection = _connectionManager.GetCollection<MongoItem>("items");
    var fdb = new FilterDefinitionBuilder<MongoItem>();
    var f = fdb.Eq(mi => mi.Id, id);
    return await collection.Find(f).FirstOrDefaultAsync();
}

How can we discover the reason and fix this issue?

Answer 1

This issue relates to CSHARP-1435, CSHARP-1515 and CSHARP-1538 bug reports, and most likely this has been fixed in the recent C# MongoDB Driver.

This issue could be related to reading number of documents being returned more than one fit in a single batch.^source

So possible solutions are:

• Make sure your MongoDB is up and running.^source
• Check MongoDB logs for any additional details.

• If connecting to group of mongod processes (see: replication), add ?connect=replicaSet to your connection string. For example:

  mongodb://db1.example.net:27017,db2.example.net:2500/?replicaSet=test&connectTimeoutMS=300000
  

  Example ConnectionStrings.config file:

  <connectionStrings>
        <add name="MongoDB" connectionString="mongodb://10.0.80.231:27017,10.0.108.31:27017/?replicaSet=groupname&amp;connectTimeoutMS=600000&amp;socketTimeoutMS=600000" />
        <add name="RedisCache" connectionString="www-redis.foo.cache.amazonaws.com:6379" />
        <add name="SqlConnection" connectionString="server=api.foo.eu-west-1.rds.amazonaws.com;database=foo;uid=sqlvpc;pwd=somepass;" providerName="System.Data.SqlClient"  />
  </connectionStrings>
  

  ^{Related: CSHARP-1160, How to include ampersand in connection string?}

  If above won't work, check: C# MongoDB Driver Ignores timeout options.

• Try upgrading MongoDB.Driver as per above bug reports.

• Try increasing connect and socket timeouts.

  Either by appending ?connectTimeoutMS=60000&socketTimeoutMS=60000 to your connection string. See: mongoDB Connection String Options.

  Alternatively changing settings in your code (e.g. connecttimeout, maxpoolsize, waitQueueSize and waitQueueTimeout). See the example here.

• Check connection string whether you're including the database properly.^source
• Increasing a delay between each query in case of rapid calls to MongoDB.^source

Please also check for MongoDB Compatibility for MongoDB C#/.NET drivers:

Answer 2

This post may help:

I figured it out. This JIRA ticket has the details.

Effectively, we've made a distinction between connecting to a standalone server and connecting directly to a replica set member, where the latter is relatively uncommon. Unfortunately, MongoLab's Single-Node settings are actually a single node replica set and this causes us to not trust it. You can fix this by appending ?connect=replicaSet to your connection string. It will force the driver to move into replica set mode and all will work.

We are going to re-consider CSHARP-1160 in light of this. Thanks so much for reporting and let me know if appending ?connect=replicaSet to your connection string doesn't work.

Answer 3

I was experiencing the same issue using driver v2.2.4. After upgrading to v2.3.0, the issue seems to have been resolved