How to limit the count of the parallel http requests in node js?

Question

I have a script which I start a several http requests inside a loop Let's say that I have to make 1000 http requests.

The thing is that I can do only one http request per IP and I have only 10 IPs.

So, after 10 parallel requests, I have to wait a response to make another one.

How do I wait without block the script one response from a http request to start another one?

My problem is if I execute a while waiting for a free IP my whole script is blocked and I do not receive any response.

Answer 1

Use the async module for this.

You can use async#eachLimit to limit the concurrent requests to 10.

var urls = [
  // a list of 100 urls
];

function makeRequest(url, callback) {
  /* make a http request */
  callback(); // when done, callback
}

async.eachLimit(urls, 10, makeRequest, function(err) {
  if(err) throw err;
});

This code will loop through the list of urls and call makeRequest for each one. It will stop at 10 concurrent requests and will not proceed with the 11th request until one of the first 10 have finished.