Removing the last character from file stream in node.js (fs module)

Question

Using node.js, I am trying to build an array of objects and write them to a file. To do this, I'm using the built in fs library.

After calling var file = fs.createWriteStream('arrayOfObjects.json'); and file.write('[') I run several asynchronous functions to eventually append objects like this:

file.write(JSON.stringify(objectToAppend) + ',\n')

I can determine when all of the objects have stopped appending, and this is where I run file.write(']') and file.end(). My problem is that adding the last comma to the end of the last object causes the JSON to be invalid.

It is very difficult to determine where and when the last object is being created due to the asynchronous nature of the script, so I was wondering if there is a way to strip or remove characters from a file-stream. If so, I could do this before adding the last ']' character.

I could do this manually, but I was hoping to pipe this to another application. The only solution I've thought about is using the fs.truncate() function, however this doesn't seem to work for file streams, and neither file.length or file.length() will give me the length of the contents because it is not a string so it's difficult to determine how or where to truncate the file.

For now I have just been adding '{}]' to the end of the array to make it valid JSON, but this empty object may cause some problems later.

Also note: the array of objects I am writing in this stream is VERY large, so I would rather not end the stream and re-open the file.

Answer 1

I'd recommend to prepend the separator instead, so that you dynamically can adjust it after the first call:

file.write('[\n')
var sep = "";
forEach(function(objectToAppen) {
    file.write(sep + JSON.stringify(objectToAppend))
    if (!sep)
        sep = ",\n";
});

Answer 2

Example using JSONStream:

var JSONStream = require('JSONStream');
var fs         = require('fs');

var jsonwriter = JSONStream.stringify();
var file       = fs.createWriteStream('arrayOfObjects.json');

// Pipe the JSON data to the file.
jsonwriter.pipe(file);

// Write your objects to the JSON stream.
jsonwriter.write({ foo : 'bar#1' });
jsonwriter.write({ foo : 'bar#2' });
jsonwriter.write({ foo : 'bar#3' });
jsonwriter.write({ foo : 'bar#4' });

// When you're done, end it.
jsonwriter.end();