XML stream writer library in Node.js

Question

I have to generate single xml file by combining and processing multiple xmls of 100MB each approximately. I am not able to find any library of node.js which can stream write the xml.

I tried below libraries

1. xml-writer
2. xmlbuilder
3. genx

All these library store the xml in memory which leads to out of memory exception.

Do you have any suggestion for appropriate library which does not store complete object in memory.

Example code for one of the library(xml-writer)

var fs = require('fs');
var XMLWriter = require('xml-writer');

var writeStream = fs.createWriteStream("myfile",{
    encoding: 'utf-8'
 });

var xw = new XMLWriter(false);
xw.startDocument('1.0', 'UTF-8').startElement(function() {
    return 'root';
});
for(var i = 0 ; i < 10000000; i++ ) {
    xw.startElement(function() {
        return 'root1';
    }).text(function() {
        return 'Some content1';
    });
    xw.endElement();
}
xw.endElement();
writeStream.write(xw.toString());
writeStream.end();

This is simple code for adding two elements in the file. But it is giving me out of memory exception. As it is creating one object and then storing complete object in the memory and writing it at once in the file.

I searched on google a lot but could not find any library which stream write the xml data to file.

Thanks,

Answer 1

xmbuilder2 has a slightly different callback API:

const fs = require('fs');
const { createCB } = require('xmlbuilder2');

const ws = fs.createWriteStream('test.xml', {
  encoding: 'utf-8',
});

const xml = createCB({
  data: (text) => {
    ws.write(text);
  },
  prettyPrint: true,
});
xml.on('end', ws.close)

xml.dec().ele('root');

for (let i = 0; i < 1000000; i++) {
  xml.ele(`root${i}`).txt('Some content1').up();
}

xml
  .up()
  .end();

Answer 2

xmlbuilder can do this using the 'callback' API. You'd just have to write to your stream in the callback function.

For example:

const xmlbuilder = require('xmlbuilder');

const xml = xmlbuilder.begin(function(chunk) { process.stdout.write(chunk); });
xml
  .dec()
  .ele('root')
;

for (let i = 0; i < 10; i++) {
  xml.ele('example' + i).up();
}

xml
  .up()  // close <root/>
  .end()
;

Some parts of the API, such as the functions for moving around and altering nodes already generated - don't work in this callback mode because they aren't cached in memory.