Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Amazon Elastic Load Balancer is not closing the connection to the server

I have an EC2 instance with Apache as a reverse proxy and ffserver as a streaming server. There is an ELB (Classic) in front of the EC2 instance which works as an SSL termination point.

Apache configuration is rather simple:

<Location "/mp3/">
    ProxyPass http://127.0.0.1:8081/ DisableReuse=On KeepAlive=Off
    ProxyPassReverse http://127.0.0.1:8081/
    SetEnv force-proxy-request-1.0.1
    SetEnv proxy-nokeepalive 1
</Location>

ffserver is used to stream live audio over the Internet. In ffserver's settings there is a MaxBandwidth option (default 1000). This setting become a problem when the connections are not closed properly. ffserver starts to respond with 503 server too busy instead of the stream's content.

If I connect to the server directly (no ELB on the road) everything works fine. If I connect over the ELB the connection won't close when I close it on the client's side (e.g. close the browser's tab).

I use the below command to check the current connections:

watch -n 2 'netstat -napt | grep 8081'

All of the connections remain in ESTABLISHED state forever (at least for 30 minutes). The default idle timeout for ELB is 60. It means that someone is receiving the stream from ffserver (connection is active).

EDIT: It looks like changing the Classic Load Balancer to the Application Load Balancer solved the problem. I don't know how to explain this behavior. Looking forward for an answer from AWS community - AWS Forums

like image 900
Mateusz Kleinert Avatar asked Apr 11 '17 15:04

Mateusz Kleinert


People also ask

How do you terminate elastic load balancing?

On the navigation pane, under LOAD BALANCING, choose Load Balancers. Select the load balancer, and then choose Actions, Delete. When prompted for confirmation, choose Yes, Delete.

Why is my load balancer not working?

If the load balancer is not responding to requests, check for the following issues: Your internet-facing load balancer is attached to a private subnet. You must specify public subnets for your load balancer. A public subnet has a route to the Internet Gateway for your virtual private cloud (VPC).

What happens when elastic load balancer fails?

An ELB is a logical entity that consists of multiple (at least 2 - one per AZ - and generally more) instances that route traffic to your back ends. If an individual ELB instance were to fail, it would be replaced automatically, much in the way autoscaling replaces failed instances.

Which of the following is not correct about elastic load balancing?

Answer: The correct answer is 2) can be enabled only only in a single availability zone. It is not true about the elastic load balancing.


2 Answers

I know this is an old thread but I recently had the same issue and the solution was to switch to ALB. Worked correctly after that.

like image 35
stefantigro Avatar answered Oct 27 '22 00:10

stefantigro


As OP shared with his edit, the issue of connections not getting closed can be resolved with a change of load balancer type. This answer focuses on why this change has such an affect?


There seems to be an issue within Classic Load Balancer (ELB). I've found the following posts with very similar issues;

  • ELB - server keeps sending data to disconnected clients for hours
  • Is there a way to lower the LBS KeepAlive?
  • AWS ELB servlet client disconnection detection

It seems the issue stems from ELB not being able to detect the client dropping from a connection. Especially when back-end is supplying some sort of data in a periodical manner, e.g. live audio stream, heartbeat, etc.

There doesn't seem to be a way to disable the keep-alive setting of load balancers, yet, somehow, only with ELB this trouble occurs.

I couldn't find exact feature that is creating this behavioural difference between ELB & ALB. I think the reason is either due to;

  • Improved Health Checks in ALB, and/or
  • an internal structural difference that is not visible to us users, that somehow keeps this issue from happening on ALB

I think the issue resolves when Application Load Balancer (ALB) is used due to said improvements and it being much more flexible.

Check here for more on the differences between ELB, ALB, and NLB


ps. AWS support forums are terrible, all the good support & tips are paywalled and stored within PMs between them & their premium customers.

like image 162
buræquete Avatar answered Oct 26 '22 23:10

buræquete