Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

concise way to transform a node.js stream?

I'm trying to convert a newline-separated JSON file to CSV (kind of a dumb task, but it doesn't matter), and am curious how I can easily do this in node.js. What I have so far is,

var csv = require('csv'), 
    fs = require('fs'),
    JSONStream = require('JSONStream');

var stream = fs.createReadStream('input.json', {encoding: 'utf8'})
    .pipe(JSONStream.parse());
csv().from(stream).to('out.csv');

This works, but doesn't let me express ordering on the keys. For example, if I have an input file like this,

{"foo": 1, "bar": 2}
{"bar": 3, "foo": -1}

I'd like to add a tranformer function,

function(line_object) { return [line_object.foo, line_object.bar]; };

so that my output will be lines 1, 2 and -1, 3, corresponding to foo being column 1 and bar being column 2, instead of the current behavior 1, 2 and 3, -1.

However, all of the examples online seem rather verbose for adding transformers. They seem to create require('stream').Transform objects, and then set the _transform function on those objects. Question: Is there a cleaner / more concise way to do this? Thank you very much, and please forgive my ignorance as this is pretty much the first node.js code I've written.

like image 895
gatoatigrado Avatar asked May 23 '26 12:05

gatoatigrado


1 Answers

event-stream might be useful:

var fs         = require('fs');
var csv        = require('csv');
var JSONStream = require('JSONStream');
var es         = require('event-stream');

es.pipeline(
  fs.createReadStream('input.json'),
  JSONStream.parse(),
  es.map(function(data, next) {
    next(null, [ data.foo, data.bar ]);
  }),
  csv().to('out.csv')
);
like image 109
robertklep Avatar answered May 25 '26 04:05

robertklep