MongoDB queries are taking 2-3 seconds from Node.js app on Heroku

Question

I am having major performance problems with MongoDB. Simple find() queries are sometimes taking 2,000-3,000 ms to complete in a database with less than 100 documents.

I am seeing this both with a MongoDB Atlas M10 instance and with a cluster that I setup on Digital Ocean on VMs with 4GB of RAM. When I restart my Node.js app on Heroku, the queries perform well (less than 100 ms) for 10-15 minutes, but then they slow down.

Am I connecting to MongoDB incorrectly or querying incorrectly from Node.js? Please see my application code below. Or is this a lack of hardware resources in a shared VM environment?

Any help will be greatly appreciated. I've done all the troubleshooting I know how with Explain query and the Mongo shell.

var Koa = require('koa'); //v2.4.1
var Router = require('koa-router'); //v7.3.0

var MongoClient = require('mongodb').MongoClient; //v3.1.3

var app = new Koa();
var router = new Router();

app.use(router.routes());

//Connect to MongoDB
async function connect() {
    try {
        var client = await MongoClient.connect(process.env.MONGODB_URI, {
            readConcern: { level: 'local' } 
        });
        var db = client.db(process.env.MONGODB_DATABASE);

        return db;
    }
    catch (error) {
        console.log(error);
    }
}

//Add MongoDB to Koa's ctx object
connect().then(db => {
    app.context.db = db;
});

//Get company's collection in MongoDB
router.get('/documents/:collection', async (ctx) => {
    try {
        var query = { company_id: ctx.state.session.company_id }; 

        var res = await ctx.db.collection(ctx.params.collection).find(query).toArray();

        ctx.body = { ok: true, docs: res };
    }
    catch (error) {
        ctx.status = 500;
        ctx.body = { ok: false };
    }
});

app.listen(process.env.PORT || 3000);

UPDATE

I am using MongoDB Change Streams and standard Server Sent Events to provide real-time updates to the application UI. I turned these off and now MongoDB appears to be performing well again.

Are MongoDB Change Streams known to impact read/write performance?

Answer 1

Change Streams indeed affect the performance of your server. As noted in this SO question.

As mentioned in the accepted answer there,

The default connection pool size in the Node.js client for MongoDB is 5. Since each change stream cursor opens a new connection, the connection pool needs to be at least as large as the number of cursors.

const mongoConnection = await MongoClient.connect(URL, {poolSize: 100});

(Thanks to MongoDB Inc. for investigating this issue.)

You need to increase your pool size to get back your normal performance.