AWS ELB Intermittent 502 gateway timeout error

Question

I have an ELB in front of a node.js application deployed on 3 EC2 instances.

I have started observing intermittent HTTP 502 Bad Gateway error.

Following is an excerpt from my access logs. There is no pattern to these 502 errors so I am unable to narrow down the cause ?

Is it an ELB issue or an app issue ?

Can access logs help me figure this out ?

This occurs for 5 requests in every 100 requests

*type*                     https    
*timestamp*                2019-05-08T14:50:11.438405Z  
*elb*                      <my-elb>
*client:port*              clientIp:port
*target:port*              targetIp:port
*request_processing_time*  0    
*target_processing_time*.  2.596    
*response_processing_time* -1   
*elb_status_code*          502  
*target_status_code*       -    
*received_bytes*           792  
*sent_bytes*               293  
*request*                  POST https://app/app-url/2.0/resourcepath/id/abc?queryParamA=abc&queryParamB=false&queryParamC=6b84c34 HTTP/1.1  
*user_agent*               Apache-CXF/3.2.5 
*ssl_cipher*               ssl-cipher
*ssl_protocol*             TLSv1.2  
*target_group_arn*         arn
*trace_id*                 traceId
*domain_name*              cool-domain-name
*chosen_cert_arn*          session-reused   
*matched_rule_priority*    0    
*request_creation_time*    2019-05-08T14:50:08.841000Z  
*actions_executed*         forward  
*redirect_url*             -    
*error_reason*             -

Answer 1

Make sure your node server keepAliveTimeout is greater than the ELB idle timeout. ELB and ALB don't like when the target machine closes the connection. To detect this you can check the elb logs, you may see response time of -1 with 502. '-1' means the target did a direct reject of the request.