How to configure AWS ELB and Nginx for WebSocket protocol? [closed]

Question

I have N-tier architecture web application in AWS. HTTP request flow order by this:

1. Nginx-ELB (public ELB, proxy to Nginx)
2. Nginx (EC2 instances in public subnet, listen on port 80, proxy to AP-ELB)
3. AP-ELB (internal ELB, proxy to AP-Server)
4. AP-Server (EC2 instances in private subnet, listen on port 80)

I want to apply WebSocket feature to this architecture. How to configure for two layer ELB and behind Nginx?

Answer 1

Use another port for ws:// protocol, because ELB don't allow listen same port in different mode (HTTP/TCP). For example: ws://Nginx-ELB:8081/ws-endpoint

That's split into two portions explain.

Nginx Section

• listen on port 80 for HTTP, then proxy to AP-ELB port 80.
• listen on port 8081 for WebSocket, then proxy to AP-ELB port 8081.

About WebSocket proxy, you can reference this configuration.

config example like this:

# Web
server {
  listen       80;
  server_name  localhost;

  charset utf-8;

  error_log /var/log/nginx/lnmnt/error.log error;
  access_log off;

  set $upstream_endpoint      <ap_elb_domain_name>;

  more_set_headers  'Cache-Control: max-age=0, no-cache, no-store';
  location / {
    proxy_connect_timeout       75;
    proxy_send_timeout          300;
    proxy_read_timeout          300;
    send_timeout                300;
    proxy_set_header        Host $host;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Host    $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_pass              $upstream_adm_endpoint;
  }
}

# WebSocket
server {
  listen  8081 proxy_protocol;
  server_name localhost;
  error_log /var/log/nginx/lnmnt/websocket.error.log error;
  access_log off;
  real_ip_header proxy_protocol;

  set $upstream_ws_endpoint   <ap_elb_domain_name>:8081;

  location / {
    proxy_set_header        Host $host;
    proxy_http_version      1.1;
    proxy_set_header        Upgrade $http_upgrade;
    proxy_set_header        Connection "upgrade";
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-Proto $scheme;
    proxy_pass              $upstream_ws_endpoint;
  }
}

ELB Section

Nginx-ELB

Create port forwarding follow this:

• 80 (HTTP) forwarding to 80 (HTTP)
• 8081 (TCP) forwarding to 8081 (TCP)

Then use your AWS CLI execute:

aws elb create-load-balancer-policy \
  --load-balancer-name Nginx-ELB \
  --policy-name EnableProxyProtocol \
  --policy-type-name ProxyProtocolPolicyType \
  --policy-attributes AttributeName=ProxyProtocol,AttributeValue=True

aws elb set-load-balancer-policies-for-backend-server \
  --load-balancer-name Nginx-ELB \
  --instance-port 8081 \
  --policy-names EnableProxyProtocol

AP-ELB

Create port forwarding follow this:

• 80 (HTTP) forwarding to 80 (HTTP)
• 8081 (TCP) forwarding to 80 (TCP)

DO NOT apply any load balancer policy for this ELB!

This section let me headache for several days. If apply same policy to both ELB, you never get the right result.

Now, enjoy your WebSocket with AWS ELB and Nginx.