Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Node.js: How to write() on Write Stream 'finish' event

I'm using Node.js streams to go through a text file line by line, make some transforms and output to a SVG file.

I am trying to write one last piece of data (</svg>) after the processing is done, however by the time the write stream emits the finish event, attempting to write() will throw Error: write after end.

Is there an elegant way I can solve this?

Note: The input file is large (around 1GB) so there's no going around the pipe() method due to its I/O and memory management.

var fs = require('fs');
var split2 = require('split2');
var through2 = require('through2');

var read_stream = fs.createReadStream('input.txt');
var write_stream = fs.createWriteStream('output.svg');

write_stream.write('<svg>');
write_stream.on('finish', function() {
  this.write('</svg>'); // doesn't work
});

read_stream
  .pipe(split2())
  .pipe(through2.obj(function(line, encoding, next) {
     this.push(line);
     next();
  }))
  .pipe(write_stream);

Solutions

Thank you Jordan & pNre for helping me figuring this out.

Solution 1 (generic)

pipe() the write stream with the end:false option and manually end() the stream.

var fs = require('fs');
var split2 = require('split2');
var through2 = require('through2');

var read_stream = fs.createReadStream('input.txt');
var write_stream = fs.createWriteStream('output.svg');

write_stream.write('<svg>');

read_stream
  .pipe(split2())
  .pipe(through2.obj(function(line, encoding, next) {
     this.push(line);
     next();
  }))
  .pipe(write_stream, { end: false });

read_stream.on('end', function() {
  write_stream.end('</svg>');
});

Solution 2 (specific to through/through2 transform streams)

through2 has a flush function that can be used to write the final data.

var fs = require('fs');
var split2 = require('split2');
var through2 = require('through2');

var read_stream = fs.createReadStream('input.txt');
var write_stream = fs.createWriteStream('output.svg');

write_stream.write('<svg>');

read_stream
  .pipe(split2())
  .pipe(through2.obj(function(line, encoding, next) {
    this.push(line);
    next();
  }, function(flush) {
    this.push('</svg>');
    flush();
  }))
  .pipe(write_stream);
like image 587
Dan Burzo Avatar asked Jan 19 '15 20:01

Dan Burzo


People also ask

How do I end a Node.js stream?

You can call the ReadStream. destroy function at any time. var fs = require("fs"); var readStream = fs.

How do I make a writable stream?

To write data to a writable stream you need to call write() on the stream instance. Like in the following example: var fs = require('fs'); var readableStream = fs. createReadStream('file1.

What is a ReadStream?

A readable stream is an abstraction for a source from which data can be consumed. An example of that is the fs. createReadStream method. A writable stream is an abstraction for a destination to which data can be written.

How do you switch between modes in readable stream mode?

One of the ways of switching the mode of a stream to flowing is to attach a 'data' event listener. A way to switch the readable stream to a flowing mode manually is to call the stream. resume method.

What is write stream in Node JS?

Write stream is primarily used to write some data to something as the data comes in. In the last blog, we looked at How to create a read stream in Node.js?. We need the knowledge on reading stream to work with write stream because after reading a file then we will be writing those chunks of data into another file using the write streams.

How to read a file sequentially and listen to events in NodeJS?

By using the fs.createWriteStream function, we created read streams to read a file’s data sequentially and listen to events from a read stream. Since Node.js WriteStreams are descendants of the Writable object, we will also listen to events to it. We have lots of control of how the write stream is created.

How to use ‘finish’ event in NodeJS?

Below examples illustrate the use of ‘finish’ event in Node.js: console.log ("Write is completed."); console.log ("program is ended."); hi program is ended. Write is completed. In the above example, writable.end () method is called before the finish event so it is emitted.

What is the use of ‘finish’ event in a writable stream?

The ‘finish’ event in a Writable Stream is emitted after the Calling of writable.end () method when all the data is being flushed to the hidden system. Return Value: If the writable.end () method is being called before then this event is emitted else its not emitted. Below examples illustrate the use of ‘finish’ event in Node.js:


Video Answer


3 Answers

It would seem that pipe closes the stream when it is finished.

The documentation at http://nodejs.org/api/stream.html states:

By default end() is called on the destination when the source stream emits end, so that destination is no longer writable. Pass { end: false } as options to keep the destination stream open.

This keeps writer open so that "Goodbye" can be written at the end.

reader.pipe(writer, { end: false });
reader.on('end', function() {
  writer.end('Goodbye\n');
});
like image 180
Jordan Honeycutt Avatar answered Sep 17 '22 15:09

Jordan Honeycutt


Have you considered creating a new stream to append the </svg> tag? through can help you with that:

var fs = require('fs');
var split2 = require('split2');
var through = require('through');

var read_stream = fs.createReadStream('input.txt');
var write_stream = fs.createWriteStream('output.svg');

write_stream.write('<svg>');
var tag = through(function write(data) {
    this.queue(data);
}, function end() {
    this.queue('</svg>');
});

read_stream.pipe(split2()).pipe(some_transform).pipe(tag).pipe(write_stream);
like image 30
pNre Avatar answered Sep 19 '22 15:09

pNre


It seems that there is an un-documented event called 'prefinish'. I have not used it though.

like image 21
advncd Avatar answered Sep 17 '22 15:09

advncd