Async calls pattern to API in Node.js

Question

Building my first "serious" Node.js project (using Express).

I need to use several calls to several REST APIs, collect all the results, massage them and return a complete JSON to the client (HTML5 + AJAX).

1. Call API A
2. Call API B
3. Call API A again (with results from B)
4. Process results from the 3 calls into a JSON
5. response.send(result)

I'm sure/hoping there's an easy pattern, or solution, or module that I just didn't google properly for :) I'd also would appreciate an opinion on where to place such operations (under 'routes'? Seperate files? etc.)

Thanks for your time!

Answer 1

The async module fits into this kind of job. Specifically, you may use the async.waterfall function.

Example:

async.waterfall([
    function(callback){
        callback(null, 'one', 'two');
    },
    function(arg1, arg2, callback){
        callback(null, 'three');
    },
    function(arg1, callback){
        // arg1 now equals 'three'
        callback(null, 'done');
    }
], function (err, result) {
   // result now equals 'done'    
});

Edit, if you have some nontrivial dependencies between the jobs, then you may use async.auto. It will determine the best order for running functions based on their requirements.

Answer 2

There are a lot of control flow libraries around. I've used Q in my previous projects which I have no complaints about, however I'll probably look into using the async library by caolan for my next project.

https://github.com/caolan/async

From what you've described above, you'd probably want to look at using the parallel function

https://github.com/caolan/async#parallel

The problem you describe can be pretty easily transferred to the parallel example in the docs

EDIT: I missed the bit about API calls being dependent. Whenever you need to pass values along the chain and control the order you'd need to use the waterfall method (see qiao's answer). If there is a case where the calls are independent, you'd use the parallel method. An example of the parallel method is below

async.parallel({
    google: function(callback){
      http.get("http://www.google.com", function(res){
        console.log("google done");
        callback(null, res.statusCode);
      })
    },
    yahoo: function(callback){
      http.get("http://www.yahoo.com", function(res){
        console.log("yahoo done");
        callback(null, res.statusCode);
      })    
    }
  },
  function(err, results) {
    if(!err){
      console.log("all done");
      console.log(results.google);
      console.log(results.yahoo);
    }else{
      console.log(err);
    }
  }
);

What this does is makes all your requests in parallel and gives you a callback when they are all done. This is where you would massage your data.

List of control flow libraries:

https://github.com/joyent/node/wiki/Modules#wiki-async-flow