Handling interdependent and/or layered asynchronous calls

Question

As an example, suppose I want to fetch a list of files from somewhere, then load the contents of these files and finally display them to the user. In a synchronous model, it would be something like this (pseudocode):

var file_list = fetchFiles(source);

if (!file_list) {
    display('failed to fetch list');

} else {
        for (file in file_list) { // iteration, not enumeration
        var data = loadFile(file);

        if (!data) {
            display('failed to load: ' + file);
        } else {
            display(data);
        }
    }
}

This provides decent feedback to the user and I can move pieces of code into functions if I so deem necessary. Life is simple.

Now, to crush my dreams: fetchFiles() and loadFile() are actually asynchronous. The easy way out is to transform them into synchronous functions. But this is not good if the browser locks up waiting for calls to complete.

How can I handle multiple interdependent and/or layered asynchronous calls without delving deeper and deeper into an endless chain of callbacks, in classic reductio ad spaghettum fashion? Is there a proven paradigm to cleanly handle these while keeping code loosely coupled?

Answer 1

Deferreds are really the way to go here. They capture exactly what you (and a whole lot of async code) want: "go away and do this potentially expensive thing, don't bother me in the meantime, and then do this when you get back."

And you don't need jQuery to use them. An enterprising individual has ported Deferred to underscore, and claims you don't even need underscore to use it.

So your code can look like this:

function fetchFiles(source) {
    var dfd = _.Deferred();

    // do some kind of thing that takes a long time
    doExpensiveThingOne({
        source: source,
        complete: function(files) {
            // this informs the Deferred that it succeeded, and passes
            // `files` to all its success ("done") handlers
            dfd.resolve(files);

            // if you know how to capture an error condition, you can also
            // indicate that with dfd.reject(...)
        }
    });

    return dfd;
}

function loadFile(file) {
    // same thing!
    var dfd = _.Deferred();

    doExpensiveThingTwo({
        file: file,
        complete: function(data) {
            dfd.resolve(data);
        }
    });

    return dfd;
}

// and now glue it together
_.when(fetchFiles(source))
.done(function(files) {
    for (var file in files) {
        _.when(loadFile(file))
        .done(function(data) {
            display(data);
        })
        .fail(function() {
            display('failed to load: ' + file);
        });
    }
})
.fail(function() {
    display('failed to fetch list');
});

The setup is a little wordier, but once you've written the code to handle the Deferred's state and stuffed it off in a function somewhere you won't have to worry about it again, you can play around with the actual flow of events very easily. For example:

var file_dfds = [];
for (var file in files) {
    file_dfds.push(loadFile(file));
}

_.when(file_dfds)
.done(function(datas) {
    // this will only run if and when ALL the files have successfully
    // loaded!
});