Why are "weird" TCP ports required for my AWS ECS app to pull from ECR?

Question

I am using ECS with NLB in front. ECS is pulling images from ECR. The thing I cannot understand is why does ECS require me to open all TCP ports to be able to pull from ECR?

2 621567429603 eni-0f5e97a3c2d51a5db 18.136.60.252 10.0.12.61 443 55584 6 13 6504 1537798711 1537798719 ACCEPT OK
2 621567429603 eni-0f5e97a3c2d51a5db 10.0.12.61 54.255.143.131 44920 443 6 13 5274 1537798711 1537798719 ACCEPT OK
2 621567429603 eni-0f5e97a3c2d51a5db 54.255.143.131 10.0.12.61 443 44952 6 13 6504 1537798711 1537798719 ACCEPT OK
2 621567429603 eni-0f5e97a3c2d51a5db 10.0.12.61 18.136.60.252 55584 443 6 15 5378 1537798711 1537798719 ACCEPT OK
2 621567429603 eni-0f5e97a3c2d51a5db 10.0.12.61 18.136.60.252 55612 443 6 15 5378 1537798711 1537798719 ACCEPT OK
2 621567429603 eni-0f5e97a3c2d51a5db 52.219.36.183 10.0.12.61 443 51892 6 19 11424 1537798711 1537798719 ACCEPT OK
2 621567429603 eni-0f5e97a3c2d51a5db 10.0.12.61 54.255.143.131 44908 443 6 14 1355 1537798711 1537798719 ACCEPT OK
2 621567429603 eni-0f5e97a3c2d51a5db 52.219.36.183 10.0.12.61 443 51912 6 31807 44085790 1537798711 1537798719 ACCEPT OK
2 621567429603 eni-0f5e97a3c2d51a5db 18.136.60.252 10.0.12.61 443 55612 6 12 6452 1537798711 1537798719 ACCEPT OK

My flow logs above. 10.0.0.0/8 is my VPC private addresses. Notice say the first time SRC: 18.136.60.252:443 is accessing 10.0.12.61:55584 why this destination port?

Then next line 2 621567429603 eni-0f5e97a3c2d51a5db 10.0.12.61 54.255.143.131 44920 443 6 13 5274 1537798711 1537798719 ACCEPT OK. Why is my ECS requesting data using source port 44920. I am asking so I know how to open the correct ports. Currently because of the ports being so random, I need to open everything

Answer 1

When it says 18.136.60.252:443 is accessing 10.0.12.61:55584, I would not say that 18.136.60.252 is "accessing" your local VPC IP. I would rather say that "18.136.60.252" is sending a response to your local VPC IP, to the random SRC port assigned by the OS to establish the TCP communication (55584), through an already ESTABLISHED TCP connection (initiated by ecs-agent in your instance).

You don't need to concentrate on which source port one should allow. You rather want to tell OS (firewall) to "let responses to get in for already established connections". In iptables is like this:

From the instance to the network, to access a remote 443 port:

iptables -A OUTPUT -o eth0 -p tcp -m multiport --dports 80,443 -j ACCEPT

From the remote host to your instance, to let responses come back:

iptables -A INPUT -i eth0 -p tcp -m multiport --sports 80,443 -m state --state RELATED,ESTABLISHED -j ACCEPT
.                                               ^ source port          ^ the rule only applies to already established connections

Here you can find some better explanations:

https://unix.stackexchange.com/q/323546/18256

https://superuser.com/a/1171322/131073

Why is my ECS requesting data using source port 44920

The OS is the one that is assigning these ports to ECS agent to be used as source port, it is just a free one, randomly selected.

---

Edit after clarifications by OP and self learning

So in AWS NACL level I should allow all ephemeral port range?

According to AWS NACL docs:

In practice, to cover the different types of clients that might initiate traffic to public-facing instances in your VPC, you can open ephemeral ports 1024-65535. However, you can also add rules to the ACL to deny traffic on any malicious ports within that range. Ensure that you place the DENY rules earlier in the table than the ALLOW rules that open the wide range of ephemeral ports.

But take into account:

You might set up network ACLs with rules similar to your security groups in order to add an additional layer of security to your VPC. (emphasis mine)

.

And at OS level do that? If I am using docker I need to do that via Dockerfile?

My recommendation is to manage this via Security Groups since they are "stateful" meaning that they track each connection that is made, automatically allowing "responses" to the ephemeral ports, without configuring those rules. For example, you can "DENY" all inbound traffic, and allow TCP 443 for outbound traffic. That does not mean that responses cannot reach the ephemeral port, they indeed can (despite the DENY all inbout traffic), because the Security Group remembers the connection. See more info here:

Security group: Is stateful: Return traffic is automatically allowed, regardless of any rules

Network ACL: Is stateless: Return traffic must be explicitly allowed by rules <-- this is answering the previous question, about ephemeral ports

Regarding the OS and iptables, I would explore first Security Groups which are easier to configure and maintain, at least for the present use case.

Answer 2

I am just adding my thoughts here based on ECS outbound network experience. Your ECS EC2 has ecs-agent running which is interacting with ECS, CloudWatch APIs continuously. ecs-agent is keep informing about EC2 host status, docker containers running and sending agent logs to above APIs.

The ecs-agent process is interacting with above AWS APIs(443) over specific intervals and that's why source port is keep changing. Here is output of my EC2 server netstat log.

command - netstat -tcp

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 ip-172-31-86-188.:47364 52.46.132.80:https      ESTABLISHED 4188/agent          
tcp        0      0 ip-172-31-86-188.:57190 52.46.132.44:https      ESTABLISHED 4188/agent 

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 ip-172-31-86-188.:57190 52.46.132.44:https      ESTABLISHED 4188/agent          
tcp        0      0 ip-172-31-86-188.:60646 52.46.128.101:https     ESTABLISHED 4188/agent    

For pulling image from ECR, your EC2 just need 443 port of ECR endpoint. All other traffic is something ecs-agent specific about cluster maintenance. Let me know your thoughts based on your EC2 TCP logs and which process is responsible for it.

Answer 3

It must be that you have the Dynamic Port Mapping configured for your ECS. It has the default ephemeral port range from 49153 through 65535, and in general, ports below 32768 are outside of the ephemeral port range, hence the randomness of host ports in TCP connections. You do have lower port values, but it must be due to your instance configuration (the default value of ip_local_port_range is most of the time is 32768 - 61000)

Please read the details on how to configure it from here, and the exact point where the port setting is set can be found here

It is possible to configure or set a single port using either set a single port for all connectivity, or edit your port range through /proc/sys/net/ipv4/ip_local_port_range, though it is not recommended to deviate from default range.