Nodejs + mikeal/Request module, how to close request or increase MaxSockets

Question

I have a Nodejs app that's designed to perform simple end-to-end testing of a large web application. This app uses the mikeal/Request and Cheerio modules to navigate, request, traverse and inspect web pages across the application.

We are refactoring some tests, and are hitting a problem when multiple request functions are called in series. I believe this may be due to the Node.js process hitting the MaxSockets limit, but am not entirely sure.

Some code...

var request = require('request');
var cheerio = require('cheerio);
var async = require('async');

var getPages_FromMenuLinks = function() {
    var pageUrl = 'http://www.example.com/app';
    async.waterfall([
        function topPageRequest(cb1) {
            var menuLinks = [];
            request(pageUrl, function(err, resp, page) {
                var $ = cheerio.load(page);
                $('div[class*="sub-menu"]').each(function (i, elem) {
                    menuLinks.push($(this).find('a').attr('href');
                });
                cb1(null, menuLinks);
            });
        }, function subMenuRequests(menuLinks, cb2) {
            async.eachSeries(menuLinks, functionv(link, callback) {
                request(link, function(err, resp, page) {
                    var $ = cheerio.load(page);
                    // do some quick validation testing of elements on the expected page
                    callback();
                });
            }, function() { cb2(null) } );
        }
    ], function () { });
};
module.export = getPages_FromMenuLinks;

Now, if I run this Node script, it runs through the first topPageRequest and starts the subMenuRequests, but then freezes after completing the request for the third sub-menu item.

It seems that I might be hitting a Max-Sockets limit, either on Node or my machine (?) -- I'm testing this on standard Windows 8 machine, running Node v0.10.26.

I've tried using request({pool:{maxSockets:25}, url:link}, function(err, resp..., but it does not seem to make any difference.

It also seems there's a way to abort the request object, if I first instantiate it (as found here). But I have no idea how I would "parse" the page, similar to what's happening in the above code. In other words, from the solution found in the link...

var theRequest = request({ ... });
theRequest.pipe(parser);
theRequest.abort();

..., how would I re-write my code to pipe and "parse" the request?

Answer 1

You can make easily thousands requests at the same time (e.g. from a single for loop) and they will be queued and terminate automatically one by one, once a particular request is served.

I think by default there are 5 sockets per domain and this limit in your case should be more than enough.

It is highly probably that your server does not handle your requests properly (e.g. on error they are not terminated and hung up indefinitely).

There are three steps you can make to find out what is going on:

1. check if you are sending proper request -- as @mattyice observed there are some bugs in your code.

2. investigate server code and the way your requests are handled there -- for me it seems that the server does not serve/terminate them in first place.

3. try to use setTimeout when sending the request. 5000ms should be a reasonable amount of time to wait. On timeout the request will be aborted with appropriate error code.

As an advice: I would recommend to use some more suitable, easier in use and more accurate tools to do your testing: e.g. phantomjs.