Node.js: http request timing out after 1 minute

Question

I am using Node.js (version 0.10.28) to make an http.request to my Ruby API for a large amount of data (21,000 rows from a PostgreSQL database). The problem is that the request seems to time out and return a 404 error after 1 minute. I know this is true as the Ruby API returns 61 seconds for the request, and I time how long the Node's request takes (results in 60 seconds and the 404). However, if I wget or use jQuery's $.ajax with a timeout of 4 minutes, I can get my 21,000 rows. So clearly it can't be a 404, as other ways of getting the data work.

I'm a little confused too because everywhere I look, the http.request isn't supposed to time out until after 2 minutes, according to:

Node.js docs

GitHub issue #1

GitHub issue #2

I have tried several things to get this working, including: setting express's middleware; listening for the socket timeout and resuming the request; and setting the timeout to 0, so there's no timeout. Unfortunately, none of these ways worked, or at least from what I've understood.

For some clarity, here's my code... setting that limit means return only 18,000 records, and that seems to be the cut off point for the API taking longer than 60 seconds:

var http = require('http');
var options = {
  path: '/api/records.json?limit=18000',
  method: 'GET',
  host: 'localhost',
  requestCert: true,
  rejectUnauthorized: false
};
var req = http.request(options, function(res) {
  var endDate = new Date();
  console.log('done', endDate - startDate);
  var output = [];
  res.on('data', function(chunk) {
    output.push(chunk);
  });
  res.on('end', function() {
    var data = output.join('');
    console.log(data);
    return {data: data, success: true};
  });
});
req.on('socket', function(socket) {
  socket.setTimeout(0);  // no timeout
  socket.on('timeout', function() {
    socket.resume();  // tried resuming the timeout
  });
});
req.end();
var startDate = new Date();
console.log('starting', startDate);

The API URL is valid and works, as I've verified that through wget and $.ajax, so how do I fix the timeout issue in Node?

UPDATE

Upon further inspection, if I take out socket.setTimeout(0); and try to get into the socket.on('timeout', I don't actually get into the callback... which is odd, but it's always 60 seconds before I get the 404

Answer 1

It's an nginx issue, as I'm using it to proxy through to my API, so it was a simple change of the proxy's configuration by setting proxy_read_timeout 180s.

Answer 2

As you said it's a nginx problem, but probably if you don't set in the nginx config the proxy parameters nginx default depends on the operative system, because in Ubuntu 16.04 I haven't any problem with a 10min timeout and without any proxy parameter, but in centos and redhat I saw the 1 min timeout restriction.

I add a configuration example because there are two parameters more than proxy_read_timeout that you need to set:

server {
    listen       80;
    server_name  myserver;

    location / {
        proxy_set_header   Host             $host;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

        proxy_connect_timeout      180s;
        proxy_send_timeout         180s;
        proxy_read_timeout         180s;

        proxy_buffer_size          4k;
        proxy_buffers              4 32k;
        proxy_busy_buffers_size    64k;
        proxy_temp_file_write_size 64k;
    }
}

For this issue the important parameters are:

proxy_connect_timeout      180s;
proxy_send_timeout         180s;
proxy_read_timeout         180s;

Answer 3

Friends have had the same problem After working for 15 days, the source of the problem is "server.headersTimeout = 7200000;" you can fix it by adding this code