I'm using Node.js streams to go through a text file line by line, make some transforms and output to a SVG file.
I am trying to write one last piece of data (</svg>
) after the processing is done, however by the time the write stream emits the finish
event, attempting to write()
will throw Error: write after end
.
Is there an elegant way I can solve this?
Note: The input file is large (around 1GB) so there's no going around the pipe()
method due to its I/O and memory management.
var fs = require('fs');
var split2 = require('split2');
var through2 = require('through2');
var read_stream = fs.createReadStream('input.txt');
var write_stream = fs.createWriteStream('output.svg');
write_stream.write('<svg>');
write_stream.on('finish', function() {
this.write('</svg>'); // doesn't work
});
read_stream
.pipe(split2())
.pipe(through2.obj(function(line, encoding, next) {
this.push(line);
next();
}))
.pipe(write_stream);
Thank you Jordan & pNre for helping me figuring this out.
pipe()
the write stream with the end:false
option and manually end()
the stream.
var fs = require('fs');
var split2 = require('split2');
var through2 = require('through2');
var read_stream = fs.createReadStream('input.txt');
var write_stream = fs.createWriteStream('output.svg');
write_stream.write('<svg>');
read_stream
.pipe(split2())
.pipe(through2.obj(function(line, encoding, next) {
this.push(line);
next();
}))
.pipe(write_stream, { end: false });
read_stream.on('end', function() {
write_stream.end('</svg>');
});
through
/through2
transform streams)through2
has a flush function that can be used to write the final data.
var fs = require('fs');
var split2 = require('split2');
var through2 = require('through2');
var read_stream = fs.createReadStream('input.txt');
var write_stream = fs.createWriteStream('output.svg');
write_stream.write('<svg>');
read_stream
.pipe(split2())
.pipe(through2.obj(function(line, encoding, next) {
this.push(line);
next();
}, function(flush) {
this.push('</svg>');
flush();
}))
.pipe(write_stream);
You can call the ReadStream. destroy function at any time. var fs = require("fs"); var readStream = fs.
To write data to a writable stream you need to call write() on the stream instance. Like in the following example: var fs = require('fs'); var readableStream = fs. createReadStream('file1.
A readable stream is an abstraction for a source from which data can be consumed. An example of that is the fs. createReadStream method. A writable stream is an abstraction for a destination to which data can be written.
One of the ways of switching the mode of a stream to flowing is to attach a 'data' event listener. A way to switch the readable stream to a flowing mode manually is to call the stream. resume method.
Write stream is primarily used to write some data to something as the data comes in. In the last blog, we looked at How to create a read stream in Node.js?. We need the knowledge on reading stream to work with write stream because after reading a file then we will be writing those chunks of data into another file using the write streams.
By using the fs.createWriteStream function, we created read streams to read a file’s data sequentially and listen to events from a read stream. Since Node.js WriteStreams are descendants of the Writable object, we will also listen to events to it. We have lots of control of how the write stream is created.
Below examples illustrate the use of ‘finish’ event in Node.js: console.log ("Write is completed."); console.log ("program is ended."); hi program is ended. Write is completed. In the above example, writable.end () method is called before the finish event so it is emitted.
The ‘finish’ event in a Writable Stream is emitted after the Calling of writable.end () method when all the data is being flushed to the hidden system. Return Value: If the writable.end () method is being called before then this event is emitted else its not emitted. Below examples illustrate the use of ‘finish’ event in Node.js:
It would seem that pipe closes the stream when it is finished.
The documentation at http://nodejs.org/api/stream.html states:
By default end() is called on the destination when the source stream emits end, so that destination is no longer writable. Pass { end: false } as options to keep the destination stream open.
This keeps writer open so that "Goodbye" can be written at the end.
reader.pipe(writer, { end: false });
reader.on('end', function() {
writer.end('Goodbye\n');
});
Have you considered creating a new stream to append the </svg>
tag?
through
can help you with that:
var fs = require('fs');
var split2 = require('split2');
var through = require('through');
var read_stream = fs.createReadStream('input.txt');
var write_stream = fs.createWriteStream('output.svg');
write_stream.write('<svg>');
var tag = through(function write(data) {
this.queue(data);
}, function end() {
this.queue('</svg>');
});
read_stream.pipe(split2()).pipe(some_transform).pipe(tag).pipe(write_stream);
It seems that there is an un-documented event called 'prefinish'
.
I have not used it though.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With