NodeJs application behind Amazon ELB throws 502

Question

We have a node application running behind Amazon Elastic Load Balancer (ELB), which randomly throws 502 errors when there are multiple concurrent requests and when each request takes time to process. Initially, we tried to increase the idle timeout of ELB to 5 minutes, but still we were getting 502 responses.

When we contacted amazon team, they said this was happening because the back-end is closing the connection with ELB after 5s.

ELB will send HTTP-502 back to your clients for the following reasons:

• The load balancer received a TCP RST from the target when attempting to establish a connection.
• The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target.
• The target response is malformed or contains HTTP headers that are not valid.
• A new target group was used but no targets have passed an initial health check yet. A target must pass one health check to be considered healthy.

We tried to set our application's keep-alive/timeouts greater than ELB idle timeout (5 min), so the ELB can be responsible for opening and closing the connections. But still, we are facing 502 errors.

js:

var http = require( 'http' );
var express = require( 'express' );
var url = require('url');
var timeout = require('connect-timeout')

const app = express();

app.get( '/health', (req, res, next) => {
  res.send( "healthy" );
});

app.get( '/api/test', (req, res, next) => {
  var query = url.parse( req.url, true ).query;
  var wait = query.wait ? parseInt(query.wait) : 1;
  setTimeout(function() {
    res.send( "Hello!" );
  }, wait );
});

var server = http.createServer(app);
server.setTimeout(10*60*1000); // 10 * 60 seconds * 1000 msecs
server.listen(80, function () {
  console.log('**** STARTING SERVER ****');
});

Answer 1

Try setting server.keepAliveTimeout to something other than the default 5s. See: https://nodejs.org/api/http.html#http_server_keepalivetimeout. Per AWS docs, you'd want this to be greater than the load balancer's idle timeout.

Note: this was added in Node v8.0.0

Also, if you're still on the Classic ELB, consider moving to the new Application Load Balancer as based on current experience this seems to have improved things for us a lot. You'll also save a few bucks if you have a lot of separate ELBs for each service. The downside could be that there's 1 point of failure for all your services. But in AWS we trust :)

Answer 2

In my case, I needed upgrade nodejs version:

https://github.com/nodejs/node/issues/27363

After that the problem was fixed.