elastic beanstalk weird nginx configuration

Question

I am trying to follow the configuration of nginx on elastic beanstalk and some things do not add up.

• The instance is opening port 80 in the security groups, so I assume all incoming traffic is coming through that port

• The nginx configuration in cat /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf states:

  server {
   listen 8080;
   location / {
      proxy_pass  http://nodejs;
      proxy_set_header   Connection "";
      proxy_http_version 1.1;
      proxy_set_header        Host            $host;
      proxy_set_header        X-Real-IP       $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
   }
   gzip on;
  }
  

  Port 8080? Where did that come from? I've tried to mess with it, this is the actual directive that is working.

• server_name is missing, yet tt doesn't matter what you put in it. If I put any value in server_name myself, this server rule will still match all requests, even the ones that are not remotely reseble the server_name value.

• While connected to the instance itself it seems both ports are being served:

  [ec2-user@ip-172-31-45-222 ~]$ sudo netstat -lnptu

  tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 22506/nginx

  tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 22506/nginx

Yet again, 8080 is never opened in the security group, so elastic load balanced is getting inside through port 80. Does traffic magically goes from 80 to 8080? Any ideas what is going on here?

Answer 1

You are forgetting to look at one part of that nginx config:

upstream nodejs {
    server 127.0.0.1:8081;
    keepalive 256;
}

That part is telling nginx to make a group of servers called nodejs as you can read about here.

8081 is the port that NodeJS is running on (if you use the sample application for instance).

You can verify this by looking at the Elastic Beanstalk logs:

-------------------------------------
/var/log/nodejs/nodejs.log
-------------------------------------
Server running at http://127.0.0.1:8081/

Then if we continue in the nginx.conf file we can see what you already posted:

server {
    listen 8080;

    location / {
        proxy_pass  http://nodejs;
        proxy_set_header   Connection "";
        proxy_http_version 1.1;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    }


}

This tells nginx to use the proxy pass module to pass all from port 8080 to our upstream group nodejs which is running on port 8081. This means that port 8081 is just for accessing it locally but port 8080 is what let's outside entities talk to the nginx which then passes stuff onto nodejs.

Some of the reasoning for not exposing NodeJS directly can be found in this StackOverflow answer.

Port 8080 is used because it is the HTTP alternate port that is "commonly used for Web proxy and caching server, or for running a Web server as a non-root user."

That explains the ports. Now the issue of ELB and how things are talking to each other.

Since the security group is only allowing access on port 80, there is an iptables rule that is setup to forward port 80 to port 8080. This allows non-root to bind to port 8080 because lower port numbers require root privileges.

You can verify this by running the following:

[ec2-user@ip-xxx-xx-xx-x ~]$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 8080

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 8080

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         

So in summary, when you load your CNAME, the load balancer is rerouting the traffic to a given instance on port 80, which is allowed through the security group, then iptables is forwarding that to port 8080, which is the port that nginx is using a proxy to pass the traffic to port 8081 which is the local port of NodeJS.

Here's a diagram:

incoming connections
-> :80             - Load Balancer
-> :80             - Security group
-> :80   -> :8080  - EC2 instance, iptables forward
-> :8080 -> :8081  - nginx, proxy pass
-> :8081           - nodejs, your app

Hopefully that helps.