Mongoose stops responding after unspecified period of time

Question

I have been tearing my hair out over this problem pretty much since I began developing my project (12 months ago!), always assuming I would find an answer before I was ready to release... unfortunately, that was not the case!

Basically, I have a fairly straightforward node.js server running on Azure connecting to a MongoLab (now MLab) database using mongoose.

The connection code looks like this:

// Connect to DB
//mongoose.set('debug', true);
mongoose.connect(envConfig.app.db, {
    server: {
        auto_reconnect: true,
        socketOptions: {
            keepAlive: 1,
            connectTimeoutMS: 30000,
            socketTimeoutMS : 30000,
        }
    },
    replset: {
        auto_reconnect: true,
        socketOptions: {
            keepAlive: 1,
            connectTimeoutMS: 30000,
            socketTimeoutMS : 30000,
        }
    }
}, function (err) {
    if (err) winstonLogger.error(err);
});
mongoose.connection.on('connecting', function () {
    console.log('Connecting to MongoDB...');
});
mongoose.connection.on('connected', function () {
    console.log('MongoDB connected!');
});
mongoose.connection.on('open', function () {
    console.log('MongoDB connection opened!');
});
mongoose.connection.on('error', function (err) {
    console.error('Error in MongoDb connection: ' + err.stack);
    winstonLogger.error(err);
    mongoose.disconnect();
});
mongoose.connection.on('disconnected', function () {
    winstonLogger.error('MongoDB disconnected!');
    mongoose.connect(envConfig.app.db, {
        server: {
            auto_reconnect: true,
            socketOptions: {
                keepAlive: 1,
                connectTimeoutMS: 30000,
                socketTimeoutMS : 30000,
            }
        },
        replset: {
            auto_reconnect: true,
            socketOptions: {
                keepAlive: 1,
                connectTimeoutMS: 30000,
                socketTimeoutMS : 30000,
            }
        }
    });
});
mongoose.connection.on('reconnected', function () {
    console.log('MongoDB reconnected!');
});
mongoose.connection.on('close', function () {
    console.log('MongoDB closed');
});

All of the extra timeout options, etc you see are part of my attempts to fix this problem (to no avail).

Here is a typical request:

AccessToken.findOne({ token: token })
            .maxTime(10000)
            .exec(function (err, accessToken) {
                // If I even got to here I would be happy
                if (err) return done(err);
                // If I could consistently get to here, my project would be finished and I would enjoy being alive again
            });

So, when I boot up my server everything works great. And it keeps working great... sometimes for days, sometimes hours, sometimes minutes. At some point, however, a request will hit this code and it will just... hang. No timeouts, no errors. Just nada. I have not found any evidence in any logs that show any evidence of what's happening. My express logger finally gives up and I get something like:

POST /api/auth/verify - - ms - -

So at this point, my only option is to restart the server, as I cannot get a db request to complete (or timeout, or show an error) for love nor money.

I have scoured the Internet for a year hoping for a solution, but everything I have tried has resulted in... nothing... the same result. I even tried running my own replica set on Azure instead of using MongoLab. Same problem. My only thinking is that it's an Azure problem with no apparent solution, but I'm really not keen on finding another host. That being said, I don't understand why the Mongoose requests don't timeout or show any error, even when I have set a maxTime...

Here's some info from MongoLab on what I thought might be the problem: http://docs.mlab.com/connecting/#known-issues, but as you can see, I have tried everything they suggest, and much more, with no success.

I am truly at a loss, and the whole situation is simultaneously brain-numbing, heartbreaking and unbelievably infuriating.

If anyone out there has ANY ideas at all, I will love you forever...

Thank you in advance!

-Luke

Answer 1

OK, so I haven't had the issue for a while now. I made a lot of small changes to the connection settings, but I think what had the biggest impact was either explicitly setting the ha (high availability) and haInterval replset options, or possibly moving the connection code to the very end of my app.js.

Here's what the final connection code looked like:

// Connect to DB
var connectionOptions = {
    replset: {
        socketOptions: {
            connectTimeoutMS: 300000, // 5 minutes
            keepAlive: 120
        },
        ha: true, // Make sure the high availability checks are on
        haInterval: 10000, // Run every 10 seconds
    }
};
//mongoose.set('debug', true);
mongoose.connect(envConfig.app.db, connectionOptions, function (err) {
    if (err) winstonLogger.error(err);
});
mongoose.connection.on('connecting', function () {
    console.log('Connecting to MongoDB...');
});
mongoose.connection.on('connected', function () {
    console.log('MongoDB connected!');
});
mongoose.connection.on('open', function () {
    console.log('MongoDB connection opened!');
});
mongoose.connection.on('error', function (err) {
    winstonLogger.error(err);
    mongoose.disconnect();
});
mongoose.connection.on('disconnected', function () {
    winstonLogger.error('MongoDB disconnected!');
    mongoose.connect(envConfig.app.db, connectionOptions, function (err) {
        if (err) winstonLogger.error(err);
    });
});
mongoose.connection.on('reconnected', function () {
    console.log('MongoDB reconnected!');
});
mongoose.connection.on('close', function () {
    winstonLogger.error('MongoDB closed');
});
if (mongoose.connection.db.serverConfig.s.replset) {
    mongoose.connection.db.serverConfig.s.replset.on('ha', function(type, data) {
        console.log('replset ha ' + type);
    });
    mongoose.connection.db.serverConfig.s.replset.on('timeout', function () {
        winstonLogger.error('MongoDB timeout');
    });
}

I also turned on the AlwaysOn option for the Azure WebApp (which didn't seem to have much effect, but I'm not switching it off now that it's working!).

It does seem like the issue had something to do with not reconnecting after a failover, but I couldn't say with 100% certainty.

Hopefully this will help someone else out who runs into the same problem... and hopefully I haven't jumped the gun only to have it stop working again in a few days (I'll be back if that happens!).

Thanks to everyone who provided a suggestion, I am ETERNALLY grateful :)

Answer 2

This sounds a great deal like an issue I had for months with the same setup (NodeJs on Azure, Mongoose, MongoLab replica set). It was "simultaneously brain-numbing, heartbreaking and unbelievably infuriating", for sure. I finally solved it a few months ago after way too many hours of frustration...

I found that it was triggered by replica set failovers. This was verified by using a test server on Azure and test replica set on MongoLab and manually triggering failovers on Mongolab (I could not repro by running the NodeJs server and MongoDB instance locally, since the problem has something to do with connection times).

The high availability checks that are supposed to help Mongoose to connect to the secondary after a failover weren't running correctly. Due to some kind of race condition, if the server takes a while to boot up after initializing the Mongoose connection, the callback to start the checks fails in mongodb-core.

What I ended up doing is moving the Mongoose connection as late into the server startup process as possible. It used to be before a pretty large file minification task, which was likely the problem.

Hope this helps!

My Mongoose settings:

// Options
mongooseOptions.replset = {
  ha: true, // Make sure the high availability checks are on
  haInterval: 5000, // Run every 5 seconds
  socketOptions : {
    ... // nothing special here
  }
}

mongoose.connect(mongodb_uri, mongooseOptions);

// Make sure the high availability checks are occuring
// Should see "replset ha start" "replset ha end" pairs every 5 secs
// Well, hopefully.
if (mongoose.connection.db.serverConfig.s.replset) {
  mongoose.connection.db.serverConfig.s.replset.on('ha', function(type, data) {
    console.log('replset ha ' + type);
  })
}