meteor - Synchronizing multiple async queries before returning?

Question

So I have a Meteor method that is supposed to tell the server to send multiple API requests to 3rd party APIs, and then combine the results of these queries into one array, which is returned to the client.

However, I can't seem to find a way for the server to wait until all the API queries have completed before returning the result.

The synchronous version of the code, which just fetches the data API call after another, goes like this:

Meteor.methods({
    fetchData: function(APILinks) {
        var data = [];
        APILinks.forEach(function(APILink) {
            var items = HTTP.get(APILink).content.items;
            items.forEach(function (item) {
                data.push(item);
            });
        });
        return items;
    }
});

This synchronous code works. However, I haven't been able to find a good way to make the API requests async. The closest I could get to a solution was to redefine the method to return the result of only one API request, and then have the client side loop through each of the API link and calling the method for each one of them. However, is there a way to wrap all these requests into one nice method that returns only when all the API requests are complete?

Answer 1

You have to use the asynchronous version of HTTP.get and collect the results using Futures.

I made up a simple example using setTimeouts to simulate HTTP requests so that you understand the principle, I advise you start from this code and replace the dummy setTimeout with your HTTP get request.

The example is a test server method which takes a number of tasks (n) as a parameter, it then launches n tasks that each compute the square of their index in index seconds.

// we use fibers which is a dependency of Meteor anyway
var Future = Npm.require("fibers/future");

Meteor.methods({
    test: function(n) {
        // build a range of tasks from 0 to n-1
        var range = _.range(n);
        // iterate sequentially over the range to launch tasks
        var futures = _.map(range, function(index) {
            var future = new Future();
            console.log("launching task", index);
            // simulate an asynchronous HTTP request using a setTimeout
            Meteor.setTimeout(function() {
                // sometime in the future, return the square of the task index
                future.return(index * index);
            }, index * 1000);
            // accumulate asynchronously parallel tasks
            return future;
        });
        // iterate sequentially over the tasks to resolve them
        var results = _.map(futures, function(future, index) {
            // waiting until the future has return
            var result = future.wait();
            console.log("result from task", index, "is", result);
            // accumulate results
            return result;
        });
        //
        console.log(results);
        return results;
    }
});

Type > Meteor.call("test",3,function(error,result){console.log(result);}); in your browser console. This will output [0,1,4] after 3 seconds.

In your server console, this will output :

// immediately :
launching task 0
launching task 1
launching task 2
// after 1 second :
result from task 0 is 0
// after 2 seconds :
result from task 1 is 1
// after 3 seconds :
result from task 2 is 4
[ 0, 1, 4 ]

The HTTP.get asynchronous version is detailed in Meteor docs :

http://docs.meteor.com/#http_call

If you want to understand better the whole Future concept, refer to the fibers docs :

https://github.com/laverdet/node-fibers