Calling multiple HTTP requests in a single HTTP request in Node.js

Question

I am trying to call multiple URL in a single URL call and push it's json response in an array and send that array in response to the end user.

My code look like this:

var express = require('express');

var main_router = express.Router();

var http = require('http');

urls = [
"http://localhost:3010/alm/build_tool",
"http://localhost:3010/alm/development_tool",
"http://localhost:3010/alm/project_architecture"];

var responses = [];

main_router.route('/')

.get(function (req, res) {

var completed_requests = 0;

for (url in urls) {

  http.get(url, function(res) {

    responses.push(res.body);

    completed_request++;

    if (completed_request == urls.length) {

        // All download done, process responses array
    }
  });
}
res.send(responses);
});

I have also tried this using npm request module. When i run this code it only return NULL or some random output that have only headers.

My aim is to call multiple URL's in a single node get request and append it's JSON output on a array and send to the end user.

Thanks

Answer 1

Here, try this code,

const async = require('async');
const request = require('request');

function httpGet(url, callback) {
  const options = {
    url :  url,
    json : true
  };
  request(options,
    function(err, res, body) {
      callback(err, body);
    }
  );
}

const urls= [
  "http://localhost:3010/alm/build_tool",
  "http://localhost:3010/alm/development_tool",
  "http://localhost:3010/alm/project_architecture"
];

async.map(urls, httpGet, function (err, res){
  if (err) return console.log(err);
  console.log(res);
});

Explanation : This code uses async and request node packages. async.map by definition takes 3 params, first one being an array, second being the iterator function you want to call with each element of that array, and the callback function, called when async.map has finished processing.

map(arr, iterator, [callback])

Produces a new array of values by mapping each value in arr through the iterator function. The iterator is called with an item from arr and a callback for when it has finished processing. Each of these callback takes 2 arguments: an error, and the transformed item from arr. If iterator passes an error to its callback, the main callback (for the map function) is immediately called with the error.

Note: All calls to iterator function are parallel.

Inside your httpGet function, you are calling request function with passed url, and explicitly telling the response format to be json. request, when finished processing, calls the callback function with three params, err - if any, res - server response, body - response body. In case there is no err from request, async.map collects the results from these callbacks as an array, and passes that array at the end to its third, callback function. Otherwise,if (err) is true, the async.map function stops the execution and calls its callback with an err.

Answer 2

I suggest to use the async library.

async.map(urls, http.get, function(err, responses){
  if (err){
    // handle error
  }
  else {
    res.send responses
  }
})

The snippet above will perform a http.get call for each of the urls in parallel, and will call your callback function with the results of all of the calls after all the responses were received.

If you want to call the urls in series, you can use async.mapSeries instead. If you want to limit the number of concurrent requests you can use async.mapLimit.