My CPU overheat when i'm using a mongo tailable cursor with .stream()

Question

I am trying to use a tailable cursor with stream. We are using mongoose and it works but when I launch my server with this code :

const listStream = ListsSub.find()
  .tailable({
    await_data: true,
    numberOfRetries: -1,
  })
  .stream();

my CPU is overheating.

Activity Monitor when code is active

Commentating the .stream() make the server runs great again.

Activity Monitor when the code is in commented

I don't really know how to do without it. Is it something with my code ? Anyone experienced the same problem ?

EDIT :

• mongoose : 4.11.3
• mongodb : 3.4.6
• node : 8.1.2
• Server and mongodb on the same machine

Answer 1

There are actually a couple of things to do here. The first notable thing is to use the .cursor() method instead of .stream() as is actually indicated in the deprecation warning issued when used otherwise:

DeprecationWarning: Mongoose: Query.prototype.stream() is deprecated in mongoose >= 4.5.0, use Query.prototype.cursor() instead

The second notable thing is that as specified in the .cursor() documentation, this now returns the "wrapped stream" interface directly from the underlying driver. So it is then recommended to use the modern .addCursorFlag() options instead of the .tailable() method from mongoose Query.

Once both of those measures are in place, I see the idle CPU for both the mongod and node processes drop to 0% in between update intervals.

This is best simulated with the following listing.

const mongoose = require('mongoose'),
      Schema = mongoose.Schema;

mongoose.Promise = global.Promise;
mongoose.set('debug',true);

const uri = 'mongodb://localhost/tailing',
      options = { useMongoClient: true };

const subSchema = new Schema({
  name: String
},{
  capped: { size: 1024, max: 1000 }
});

const Sub = mongoose.model('Sub', subSchema);

function log(data) {
  console.log(JSON.stringify(data, undefined, 2))
}

(async function() {

  try {

    const conn = await mongoose.connect(uri,options);

    //await Sub.remove({});

    await Sub.insertMany(Array(50).fill(1).map((e,i) => ({ name: i+1 })));

    let stream = Sub.find()
      .cursor()
      .addCursorFlag('tailable',true)
      .addCursorFlag('awaitData',true);
      /*
      .tailable({
        await_data: true,
        numberOfRetries: -1
      })
      .cursor();
      */

    stream.on('data',function(data) {
      log(data);
    });

    let counter = 50;


    setInterval(async function() {
      counter++;
      await Sub.insertMany({ name: counter });
    },10000);

  } catch(e) {
    console.log(e);
  } finally {
    //mongoose.disconnect();
  }

})();

Plain old top output, captured as an actual write takes place:

top - 21:38:29 up 12 days,  1:23,  3 users,  load average: 0.06, 0.03, 0.04
Tasks: 116 total,   2 running, 114 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.3 sy,  0.0 ni, 98.6 id,  0.7 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  2045968 total,   207452 free,   813908 used,  1024608 buff/cache
KiB Swap:  2097148 total,  2097124 free,       24 used.  1028156 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 1257 mongodb   20   0 1946896 487336  34892 S  0.7 23.8 130:37.67 mongod
28233 neillunn  20   0 1021460  41920  22996 S  0.3  2.0   0:00.67 node
30956 neillunn  20   0  101472   4384   3352 S  0.3  0.2   0:20.95 sshd