How can I write blocking in stdout with node.js?

Question

I'm writing a node.js application which stdout is piped to a file. I'm writing everything with console.log. After a while my Application reaches the 1GB Limit and stops. The interesting thing is, that if I use console.error instead of console.log, the memory usage keeps low and the programm runs fine. So it looks like node.js can't flush the stdout stream and everything is kept in memory. I wanna keep stderr free for errors.

My Question is:

Is there a way to write blocking into stdout? Or at least, can I write with a callback to stdout, so I can ensure I'm writing not too much?

thx!

Answer 1

Write using process.stdout.write, the return value is whether data got buffered. If it's true, continue writing when process.stdout emits the drain event.

If you want your code to look sync, use streamlinejs as described here: Node.js stdout flush

Answer 2

The problem (exploding memory usage) probably occurs because your program is creating output faster than it can be display. Therefore you want to throttle it. Your question requests "synchronous output", but actually the problem can be solved by using purely "asynchronous"(*) code.

(* NOTE: In this post the term "asynchronous" is used in the "javascript-single-thread" sense. That differs from the conventional "multi-thread" sense which an entirely different kettle of fish).

This answer shows how "asynchronous" code can be used with Promises to prevent memory usage from exploding by "pausing" (as opposed to blocking) execution until the write output has been successfully flushed. This answer also explains how an asynchronous code solution can be advantageous compared to a synchronous code solution.

Q: "Paused" sounds like "blocked", and how can asynchronous code possibly "block"? That's an oxymoron!

A: It works because the the javascript v8 engine pauses (blocks) execution of only the single code slice await an asynchronous promise to complete, while permitting other code slices to execute in the meantime.

Here is an asynchronous write function (adapted from here).

async function streamWriteAsync(
  stream,
  chunk,
  encoding='utf8') {
  return await new Promise((resolve, reject) => {
    const errListener = (err) => {
      stream.removeListener('error', errListener);
      reject(err);
    };
    stream.addListener('error', errListener);
    const callback = () => {
      stream.removeListener('error', errListener);
      resolve(undefined);
    };
    stream.write(chunk, encoding, callback);
  });
}

It can be called from an asynchrous function in your source code, e.g.
case 1

async function main() {
  while (true)
    await streamWriteAsync(process.stdout, 'hello world\n')
}
main();

Where main() is the only function called from the top level. The memory usage will not explode as it would if calling console.log('hello world');.

More context is required to clearly see the advantage over a true synchronous write:
case 2

async function logger() {
  while (true)
    await streamWriteAsync(process.stdout, 'hello world\n')
}
const snooze = ms => new Promise(resolve => setTimeout(resolve, ms));
function allowOtherThreadsToRun(){
  return Promise(resolve => setTimeout(resolve, 0));
}
async function essentialWorker(){
  let a=0,b=1;
  while (true) {
    let tmp=a; a=b; b=tmp;
    allowOtherThreadsToRun();
  }
}
async function main(){
  Promise.all([logger(), essentialWorker()])  
}
main();

Running the above code (case 2) would show that the memory use is still not exploding (same as case 1) because the logger associated slice was paused, but the CPU usage was still because as the essentialWorker slice was not paused - which is good (think COVID).

In comparison, a synchronous solution would also block the essentialWorker.

What happens with multiple slices calling streamWrite?
case 3

async function loggerHi() {
  while (true)
    await streamWriteAsync(process.stdout, 'hello world\n')
}
async function loggerBye() {
  while (true)
    await streamWriteAsync(process.stdout, 'goodbye world\n')
}
function allowOtherThreadsToRun(){
  return Promise(resolve => setTimeout(resolve, 0));
}
async function essentialWorker(){
  let a=0,b=1;
  while (true) {
    let tmp=a; a=b; b=tmp;
    allowOtherThreadsToRun();
  }
}
async function main(){
  Promise.all([loggerHi(), loggerBye(), essentialWorker()])  
}
main();

In this case (case 3), the memory usage is bound, and the essentialWorker CPU usage is high, the same as in case 2. Individual lines of hello world and goodbye world would remain atomic, but the lines would not alternate cleanly, e.g.,

...
hello world 
hello world 
goodbye world 
hello world 
hello world 
...

could appear.