Sending API calls in batches

Question

I'm currently trying to simulate half a million IoT devices to push payload to Azure IoT Hub using nodejs. Since node is multi-threaded in nature, its flooding iot hub with data and i am getting network errors.

I also tried async/await method but that is taking a lot of time to push data to IoT Hub.

Is there a way to only run 100 calls in parallel, wait for all of them to complete and then run the next 100 in node?

Much appreciated!

Answer 1

Build your batches as a nested array of Promises, then use Promise.all on each batch in a loop that awaits for each Promise.all to resolve.

// This is a mock request function, could be a `request` call 
// or a database query; whatever it is, it MUST return a Promise.
const sendRequest = () => {
  return new Promise((resolve) => {
    setTimeout(() => {
      console.log('request sent')
      resolve()
    }, 1000)
  })
}

// 5 batches * 2 requests = 10 requests.
const batches = Array(5).fill(Array(2).fill(sendRequest))

;(async function() {
  for (const batch of batches) {
    try {
      console.log('-- sending batch --')
      await Promise.all(batch.map(f => f()))  
    } catch(err) {
      console.error(err)
    }
  }
})()

Answer 2

You can readily use bluebird Promise's map with concurrency option. This processes the max records as mentioned in the concurrency, before picking up the next batch. example :

Promise.map([], {concurrency : 100})

Answer 3

If you are using lodash you can make it a bit easier by using chunk which will divide an array into chunks of provided max size

So in your case you can use it like this

variable calls (array of 550 lets say)

const batchCalls = _.chunk(calls, 100);

for (const batchCall of batchCalls) {
  await Promise.all(batchCall.map(call => call())) // makes a hundred calls in series
}

Answer 4

limited-request-queue could be used to queue the request. There are options to set the Maximum number of connections at any given time. Below is the code we used to send 5 request every second. Also there will only be 5 request sent at any given time.

limited-request-queue

/* 
   Request passed to Targer App (5 requests per seconds) 
   Get the response for each request and passed the response to Source App
   maxSockets: The maximum number of connections allowed at any given time. A value of 0 will prevent anything from going out. A value of Infinity will provide no concurrency limiting.
   maxSocketsPerHost:The maximum number of connections per host allowed at any given time. A value of 0 will prevent anything from going out. A value of Infinity will provide no per-host concurrency limiting.
   rateLimit: The number of milliseconds to wait before each maxSocketsPerHost
   */
var queue1 = new RequestQueue({'maxSockets': 5, 'maxSocketsPerHost': 5, 'rateLimit': 1000}, {
  item: function(input, done) {
      request(input.url, function(error, response) {
        input.res.send(response.body);
        done();
      });
  },
  end: function() {
      console.log("Queue  1 completed!");
  }
});

//To queue request - A for loop could be used to send multiple request
queue1.enqueue({'url': ''});