Repetitive Node.js requests to MongoDB slow down eventually

Question

I have data going into a MongoDB collection rmc and it's being upserted, i.e. I have one point with the latest latitude and longitude for my device.

From Node.js, I'd like to query that collection every 100 ms (to simulate real time) and update a map with the updated latitude / longitude.

I get good performance at first, but right after having updated data in my collection or just a while, the performance starts getting really bad.

What am I doing wrong? Could I do things in a better way? I can't seem to figure out if MongoDB or Node, or Mongoose.

The user goes to index.html, which fetches an HTML page. Within the HTML, I request a page every 100 ms:

function updateData() {
  $.getJSON("/data", function(json) {
    dosomething()
    interval = setTimeout(updateData, 100);
  })
};

updateData();

And my index.js:

/* GET home page. */
router.get('/', function(req, res, next) {
  res.render('index', { title: "test" });
});

router.get('/data', function(req, res, next) {
  var db = req.db;
  Json.find({}).select({}).lean().exec(function(e,docs){
    res.json(docs);
  });
});

Things are good and all of a sudden, there is huge delay being experienced:

GET /data 304 1.644 ms - -
GET /data 304 1.738 ms - -
GET /data 304 1.685 ms - -
GET /data 304 1.693 ms - -
GET /data 304 1.624 ms - -
GET /data 304 1.645 ms - -
GET /data 304 1.873 ms - -
GET /data 304 1.607 ms - -
GET /data 304 1.638 ms - -
GET /data 304 1.610 ms - -
GET /data 304 1.734 ms - -
GET /data 304 1.736 ms - -
GET /data 304 1.660 ms - -
GET /data 304 1.634 ms - -
GET /data 304 15.265 ms - -
GET /data 304 10.535 ms - -
GET /data 304 1.740 ms - -
GET /data 304 70.184 ms - -
GET /data 304 69.037 ms - -
GET /data 304 58.620 ms - -
GET /data 304 75.053 ms - -
GET /data 304 72.292 ms - -
GET /data 304 92.447 ms - -
GET /data 304 95.270 ms - -
GET /data 304 448.057 ms - -
GET /data 304 567.309 ms - -
GET /data 304 683.199 ms - -
GET /data 304 731.952 ms - -
GET /data 304 1102.502 ms - -
GET /data 304 1770.029 ms - -
GET /data 304 1051.307 ms - -
GET /data 304 1059.791 ms - -

Answer 1

There might be different ways for profiling the node application to find out where the issue come from. Using native node you can try running you app:

 node --prof <entrypoint>.js

Then try to run your test to allow node to gather some data, when finished a file will be generated that has to be converted to something with a better human friendly output

node --prof-process <file_generate>.log > processed.txt

Getting into processed.txt you can find which parts of your code spend more CPU time so can point you to potentially the place where the issue might come from.

For more detailed info, there is a quite good entry in node docs about analyzing profiling files https://nodejs.org/en/docs/guides/simple-profiling/

Answer 2

I think you are piping the output to many functions which don't do much. Also, if the output is small then it seems okay to send all the data as response. Try modifying your code to

const docs = await Json.find({})
res.json(docs)

You don't need lean() or select(). They are doing nothing