Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Pausing readline in Node.js

Tags:

node.js

Consider the code below ... I am trying to pause the stream after reading the first 5 lines:

var fs          = require('fs');
var readline    = require('readline');
var stream      = require('stream');
var numlines    = 0;
var instream    = fs.createReadStream("myfile.json");
var outstream   = new stream;
var readStream = readline.createInterface(instream, outstream);
readStream.on('line', function(line){
  numlines++;
  console.log("Read " + numlines + " lines");
  if (numlines >= 5) {
    console.log("Pausing stream");
    readStream.pause();
  }
});

The output (copied next) suggests that it keeps reading lines after the pause. Perhaps readline has queued up a few more lines in the buffer, and is feeding them to me anyway ... this would make sense if it continues to read asynchronously in the background, but based on the documentation, I don't know what the proper behavior should be. Any recommendations on how to achieve the desired effect?

Read 1 lines
Read 2 lines
Read 3 lines
Read 4 lines
Read 5 lines
Pausing stream
Read 6 lines
Pausing stream
Read 7 lines
like image 583
Michael D. Moffitt Avatar asked Jan 24 '14 19:01

Michael D. Moffitt


People also ask

How do I pause NodeJS?

One way to delay execution of a function in NodeJS is to use the seTimeout() function. Just put the code you want to delay in the callback. For example, below is how you can wait 1 second before executing some code.

What is createInterface in NodeJS?

The method createInterface() takes two parameters – the input stream and output stream – to create a readline interface. The third parameter is used for autocompletion and is mostly initialized as NULL .

What is readline in NodeJS?

Readline Module in Node.js allows the reading of input stream line by line. This module wraps up the process standard output and process standard input objects. Readline module makes it easier for input and reading the output given by the user.

How use readline module in NodeJS?

The readline module in NodeJS provides you with a way to read data stream from a file or ask your user for an input. To use the module, you need to import it to your JavaScript file as follows: const readline = require('readline');


2 Answers

So, it turns out that the readline stream tends to "drip" (i.e., leak a few extra lines) even after a pause(). The documentation does not make this clear, but it's true.

If you want the pause() toggle to appear immediate, you'll have to create your own line buffer and accumulate the leftover lines yourself.

like image 106
Michael D. Moffitt Avatar answered Oct 02 '22 10:10

Michael D. Moffitt


Somewhat unintuitively, the pause methods does not stop queued up line events:

Calling rl.pause() does not immediately pause other events (including 'line') from being emitted by the readline.Interface instance.

There is however a 3rd-party module named line-by-line where pause does pause the line events until it is resumed.

var LineByLineReader = require('line-by-line'),
    lr = new LineByLineReader('big_file.txt');

lr.on('error', function (err) {
  // 'err' contains error object
});

lr.on('line', function (line) {
  // pause emitting of lines...
  lr.pause();

  // ...do your asynchronous line processing..
  setTimeout(function () {

      // ...and continue emitting lines.
      lr.resume();
  }, 100);
});

lr.on('end', function () {
  // All lines are read, file is closed now.
});

(I have no affiliation with the module, just found it useful for dealing with this issue.)

like image 35
Alexander O'Mara Avatar answered Oct 02 '22 12:10

Alexander O'Mara