throttle requests in Node.js

Question

I have an array. I can loop over it with the foreach method.

data.forEach(function (result, i) {

     url = data[i].url;

     request(url);

});

The request function is making a http request to the given url. However making all these requests at the same time leads to all sorts of problems.

So I thought I should slow things down by introducing some-sort of timer.

But I have no idea how will be able to combine a forach loop with setTimeOut/setInterval

Please note am doing this on the server (nodejs) rather on the browser.

Thanks for you help.

Answer 1

As your problem is global, you should adjust your request function to have only 5 request running at a time - using a global, static counter. If your request was before something like

function request(url, callback) {
    ajax(url, callback);
}

now use something like

var count = 0;
var waiting = [];
function request(url, callback) {
    if (count < 5) {
        count++;
        ajax(url, function() {
            count--;
            if (waiting.length)
                request.apply(null, waiting.shift());
            callback.apply(this, arguments);
        });
    } else
        waiting.push(arguments);
}

Answer 2

data.forEach(function (result, i) {

     url = data[i].url;

     setTimeout(
         function () {
              request(url);
         }, 
         1000 * (i + 1) // where they will each progressively wait 1 sec more each
     );

 });