How to avoid long nesting of asynchronous functions in Node.js

Question

Interesting observation. Note that in JavaScript you can normally replace inline anonymous callback functions with named function variables.

The following:

http.createServer(function (req, res) {
   // inline callback function ...

   getSomeData(client, function (someData) {
      // another inline callback function ...

      getMoreData(client, function(moreData) {
         // one more inline callback function ...
      });
   });

   // etc ...
});

Could be rewritten to look something like this:

var moreDataParser = function (moreData) {
   // date parsing logic
};

var someDataParser = function (someData) {
   // some data parsing logic

   getMoreData(client, moreDataParser);
};

var createServerCallback = function (req, res) {
   // create server logic

   getSomeData(client, someDataParser);

   // etc ...
};

http.createServer(createServerCallback);

However unless you plan to reuse to callback logic in other places, it is often much easier to read inline anonymous functions, as in your example. It will also spare you from having to find a name for all the callbacks.

In addition note that as @pst noted in a comment below, if you are accessing closure variables within the inner functions, the above would not be a straightforward translation. In such cases, using inline anonymous functions is even more preferable.

Kay, simply use one of these modules.

• flow-js
• funk
• futures
• groupie
• node-continuables
• Slide
• Step
• node-inflow
• async.js
• async

It will turn this:

dbGet('userIdOf:bobvance', function(userId) {
    dbSet('user:' + userId + ':email', '[email protected]', function() {
        dbSet('user:' + userId + ':firstName', 'Bob', function() {
            dbSet('user:' + userId + ':lastName', 'Vance', function() {
                okWeAreDone();
            });
        });
    });
});

Into this:

flow.exec(
    function() {
        dbGet('userIdOf:bobvance', this);

    },function(userId) {
        dbSet('user:' + userId + ':email', '[email protected]', this.MULTI());
        dbSet('user:' + userId + ':firstName', 'Bob', this.MULTI());
        dbSet('user:' + userId + ':lastName', 'Vance', this.MULTI());

    },function() {
        okWeAreDone()
    }
);

For the most part, I'd agree with Daniel Vassallo. If you can break up a complicated and deeply nested function into separate named functions, then that is usually a good idea. For the times when it makes sense to do it inside a single function, you can use one of the many node.js async libraries available. People have come up with lots of different ways to tackle this, so take a look at the node.js modules page and see what you think.

I've written a module for this myself, called async.js. Using this, the above example could be updated to:

http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/html'});
  async.series({
    someData: async.apply(getSomeDate, client),
    someOtherData: async.apply(getSomeOtherDate, client),
    moreData: async.apply(getMoreData, client)
  },
  function (err, results) {
    var html = "<h1>Demo page</h1>";
    html += "<p>" + results.someData + "</p>";
    html += "<p>" + results.someOtherData + "</p>";
    html += "<p>" + results.moreData + "</p>";
    res.write(html);
    res.end();
  });
});

One nice thing about this approach is that you can quickly change your code to fetch the data in parallel by changing the 'series' function to 'parallel'. What's more, async.js will also work inside the browser, so you can use the same methods as you would in node.js should you encounter any tricky async code.

Hope that's useful!

You could use this trick with an array rather than nested functions or a module.

Far easier on the eyes.

var fs = require("fs");
var chain = [
    function() { 
        console.log("step1");
        fs.stat("f1.js",chain.shift());
    },
    function(err, stats) {
        console.log("step2");
        fs.stat("f2.js",chain.shift());
    },
    function(err, stats) {
        console.log("step3");
        fs.stat("f2.js",chain.shift());
    },
    function(err, stats) {
        console.log("step4");
        fs.stat("f2.js",chain.shift());
    },
    function(err, stats) {
        console.log("step5");
        fs.stat("f2.js",chain.shift());
    },
    function(err, stats) {
        console.log("done");
    },
];
chain.shift()();

You can extend the idiom for parallel processes or even parallel chains of processes:

var fs = require("fs");
var fork1 = 2, fork2 = 2, chain = [
    function() { 
        console.log("step1");
        fs.stat("f1.js",chain.shift());
    },
    function(err, stats) {
        console.log("step2");
        var next = chain.shift();
        fs.stat("f2a.js",next);
        fs.stat("f2b.js",next);
    },
    function(err, stats) {
        if ( --fork1 )
            return;
        console.log("step3");
        var next = chain.shift();

        var chain1 = [
            function() { 
                console.log("step4aa");
                fs.stat("f1.js",chain1.shift());
            },
            function(err, stats) { 
                console.log("step4ab");
                fs.stat("f1ab.js",next);
            },
        ];
        chain1.shift()();

        var chain2 = [
            function() { 
                console.log("step4ba");
                fs.stat("f1.js",chain2.shift());
            },
            function(err, stats) { 
                console.log("step4bb");
                fs.stat("f1ab.js",next);
            },
        ];
        chain2.shift()();
    },
    function(err, stats) {
        if ( --fork2 )
            return;
        console.log("done");
    },
];
chain.shift()();

I like async.js a lot for this purpose.

The issue is solved by waterfall command:

waterfall(tasks, [callback])

Runs an array of functions in series, each passing their results to the next in the array. However, if any of the functions pass an error to the callback, the next function is not executed and the main callback is immediately called with the error.

Arguments

tasks - An array of functions to run, each function is passed a callback(err, result1, result2, ...) it must call on completion. The first argument is an error (which can be null) and any further arguments will be passed as arguments in order to the next task. callback(err, [results]) - An optional callback to run once all the functions have completed. This will be passed the results of the last task's callback.

Example

async.waterfall([
    function(callback){
        callback(null, 'one', 'two');
    },
    function(arg1, arg2, callback){
        callback(null, 'three');
    },
    function(arg1, callback){
        // arg1 now equals 'three'
        callback(null, 'done');
    }
], function (err, result) {
    // result now equals 'done'    
});

As for the req,res variables, they will be shared within the same scope as function(req,res){} which enclosed the whole async.waterfall call.

Not only so, async is very clean. What I means is that I change a lot of cases like this:

function(o,cb){
    function2(o,function(err, resp){
        cb(err,resp);
    })
}

To first:

function(o,cb){
    function2(o,cb);
}

Then to this:

function2(o,cb);

Then to this:

async.waterfall([function2,function3,function4],optionalcb)

It also allows many premade functions prepared for async to be called from util.js very fast. Just chain up what you want to do, make sure o,cb is universally handled. This speeds up the whole coding process a lot.