Transforming JSON in a node stream with a map or template

Question

I'm relatively new to Javascript and Node and I like to learn by doing, but my lack of awareness of Javascript design patterns makes me wary of trying to reinvent the wheel, I'd like to know from the community if what I want to do is already present in some form or another, I'm not looking for specific code for the example below, just a nudge in the right direction and what I should be searching for.

I basically want to create my own private IFTTT/Zapier for plugging data from one API to another.

I'm using the node module request to GET data from one API and then POST to another.

request supports streaming to do neat things like this:

request.get('http://example.com/api')
  .pipe(request.put('http://example.com/api2'));

In between those two requests, I'd like to pipe the JSON through a transform, cherry picking the key/value pairs that I need and changing the keys to what the destination API is expecting.

request.get('http://example.com/api')
  .pipe(apiToApi2Map)
  .pipe(request.put('http://example.com/api2'));

Here's a JSON sample from the source API: http://pastebin.com/iKYTJCYk

And this is what I'd like to send forward: http://pastebin.com/133RhSJT

The transformed JSON in this case takes the keys from the value of each objects "attribute" key and the value from each objects "value" key.

So my questions:

• Is there a framework, library or module that will make the transform step easier?

• Is streaming the way I should be approaching this? It seems like an elegant way to do it, as I've created some Javascript wrapper functions with request to easily access API methods, I just need to figure out the middle step.

• Would it be possible to create "templates" or "maps" for these transforms? Say I want to change the source or destination API, it would be nice to create a new file that maps the source to destination key/values required.

Hope the community can help and I'm open to any and all suggestions! :) This is an Open Source project I'm working on, so if anyone would like to get involved, just get in touch.

Answer 1

Yes you're definitely on the right track. There are two stream libs I would point you towards, through which makes it easier to define your own streams, and JSONStream which helps to convert a binary stream (like what you get from request.get) into a stream of parsed JSON documents. Here's an example using both of those to get you started:

var through = require('through');
var request = require('request');
var JSONStream = require('JSONStream');
var _ = require('underscore');

// Our function(doc) here will get called to handle each
// incoming document int he attributes array of the JSON stream
var transformer = through(function(doc) {
    var steps = _.findWhere(doc.items, {
        label: "Steps"
    });
    var activeMinutes = _.findWhere(doc.items, {
        label: "Active minutes"
    });
    var stepsGoal = _.findWhere(doc.items, {
        label: "Steps goal"
    });

    // Push the transformed document into the outgoing stream
    this.queue({
        steps: steps.value,
        activeMinutes: activeMinutes.value,
        stepsGoal: stepsGoal.value
    });
});

request
    .get('http://example.com/api')
    // The attributes.* here will split the JSON stream into chunks
    // where each chunk is an element of the array
    .pipe(JSONStream.parse('attributes.*'))
    .pipe(transformer)
    .pipe(request.put('http://example.com/api2'));

Answer 2

As Andrew pointed out there's through or event-stream, however I made something even easier to use, scramjet. It works the same way as through, but it's API is nearly identical to Arrays, so you can use map and filter methods easily.

The code for your example would be:

DataStream
   .pipeline(
       request.get('http://example.com/api'),
       JSONStream.parse('attributes.items.*')
   )
   .filter((item) => item.attibute)  // filter out ones without attribute
   .reduce((acc, item) => {
       acc[item.attribute] = item.value;
       return acc;
   .then((result) => request.put('http://example.com/api2', result))
;

I guess this is a little easier to use - however in this example you do accumulate the data into an object - so if the JSON's are actually much longer than this, you may want to turn it back into a JSONStream again.