Sockets don't appear to be closing when using Node.js http.get

Question

OS X 10.8.3

Node 0.10.0

I'm using the 'http' module to make requests of the Facebook graph API.

Here are the options that I pass to 'http.get':

var options = {host: 'graph.facebook.com',
               port: 80,
               path: '/' + fb_id + '/picture'};  //fb_id is a Facebook user identifier

My code looks like this:

http.get(options,
  function(res) {
    ...some stuff...
    DONE(RESULT);  //DONE is a callback function
  }).on('error', function(e) {
       ...some error handling...
});

What I observe is that I can only do as many requests as the value of http.globalAgent.maxSockets. Once I reach that many requests, the next call to http.get never (apparently) connects. I've verified that I'm not getting errors on the requests.

It's as though the sockets are not being closed after the response comes in.

Is there something I need to do as part of the response handler to ensure that the socket is closed?

Are these sockets not closing because of the default keepalive behavior?

How should I proceed to debug this?

Answer 1

Try setting agent: false in the options. Default behaviour is indeed to keep connections open for HTTP keep-alive:

var options = {host: 'graph.facebook.com',
               port: 80,
               path: '/' + fb_id + '/picture',
               agent: false};

Answer 2

Node's http module states that agent defaults to global agent: http://nodejs.org/api/http.html#http_http_globalagent, which means that keep-alive is shared regardless of the module that origins the request.

BTW, responding to Wes' comment of Apr9'13 at 20:47: it does not matter how many times you load a node module, it will be loaded only once and share by all the modules.

What you're experiencing is a pool exhaustion problem. The simplest way to avoid it is to use a new agent (http://nodejs.org/api/http.html#http_class_http_agent) with your desired maxSockets. Remember that the agent you create can be shared between modules if you place it in an export of that module (modules in node are stateful!!!) .

Answer 3

I experienced the same behavior, except that my connections were finally reused after a timeout period. Check if the connections are reused after a certain period of time (couple of minutes), and also check if response headers contain 'Connection: keep-alive'.

If that's the case a possible solution would be to use the 'Connection: Close' header instead of keep-alive, that way pooled connections could be reused earlier as in the usual setup. I am not sure if this leads to any performance issues using the facebook endpoints.

var options = {host: 'graph.facebook.com',
               port: 80,
               path: '/' + fb_id + '/picture',
               headers: { 'Connection':'Close' }
};

For me using agent:false did not work because the vast number of requests I sent exhausted server resources.