Can't connect to port 80 on Google Cloud Compute instance despite firewall rule

Question

In summary, although I've set a firewall rule that allows tcp:80, my GCE instance, which is on the "default" network, is not accepting connections to port 80. It appears only port 22 is open on my instance. I can ping it, but can't traceroute to it in under 64 hops.

What follows is my investigation that led me to those conclusions.

gcloud beta compute firewall-rules list

NAME                    NETWORK  DIRECTION  PRIORITY  ALLOW                         DENY
default-allow-http      default  INGRESS    1000      tcp:80
default-allow-https     default  INGRESS    1000      tcp:443
default-allow-icmp      default  INGRESS    65534     icmp
default-allow-internal  default  INGRESS    65534     tcp:0-65535,udp:0-65535,icmp
default-allow-rdp       default  INGRESS    65534     tcp:3389
default-allow-ssh       default  INGRESS    65534     tcp:22
temp                    default  INGRESS    1000      tcp:8888


gcloud compute instances list
NAME   ZONE        MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
ssrf3  us-west1-c  f1-micro      true         10.138.0.4   35.197.33.182  RUNNING


gcloud compute instances describe ssrf3
...
name: ssrf3
networkInterfaces:
- accessConfigs:
  - kind: compute#accessConfig
    name: external-nat
    natIP: 35.197.33.182
    type: ONE_TO_ONE_NAT
  kind: compute#networkInterface
  name: nic0
  network: https://www.googleapis.com/compute/v1/projects/hack-170416/global/networks/default
  networkIP: 10.138.0.4
  subnetwork: https://www.googleapis.com/compute/v1/projects/hack-170416/regions/us-west1/subnetworks/default
...
tags:
  fingerprint: 6smc4R4d39I=
  items:
  - http-server
  - https-server

I ssh into 35.197.33.182 (which is the ssrf3 instance) and run:

sudo nc -l -vv -p 80

On my local machine, I run:

nc 35.197.33.182 80 -vv
hey

but nothing happens. So I try to ping the host. That looks healthy:

ping 35.197.33.182 
PING 35.197.33.182 (35.197.33.182): 56 data bytes
64 bytes from 35.197.33.182: icmp_seq=0 ttl=57 time=69.172 ms
64 bytes from 35.197.33.182: icmp_seq=1 ttl=57 time=21.509 ms

Traceroute quits after 64 hops, without reaching the 35.197.33.182 destination.

So I check which ports are open with nmap:

nmap 35.197.33.182

Starting Nmap 7.12 ( https://nmap.org ) at 2017-06-18 16:39 PDT
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.06 seconds



nmap 35.197.33.182 -Pn

Starting Nmap 7.12 ( https://nmap.org ) at 2017-06-18 16:39 PDT
Nmap scan report for 182.33.197.35.bc.googleusercontent.com (35.197.33.182)
Host is up (0.022s latency).
Not shown: 999 filtered ports
PORT   STATE SERVICE
22/tcp open  ssh

Nmap done: 1 IP address (1 host up) scanned in 6.84 seconds

… even when I’m running nc -l -p 80 on 35.197.33.182.

Answer 1

Ensure that VM level firewall is not intervening. For example, Container-Optimized OS is a bit special in comparison to all other default images:

By default, the Container-Optimized OS host firewall allows only outgoing connections, and accepts incoming connections only through the SSH service. To accept incoming connections on a Container-Optimized OS instance, you must open the ports your services are listening on.

https://cloud.google.com/container-optimized-os/docs/how-to/firewall

Answer 2

After lots of trail and error, the following worked for me on ubuntu-1404-trusty-v20190514 , with a nodejs app listening on port 8080. Accept port 80 and 8080, and then redirect 80 to 8080.

sudo iptables -A INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -i eth0 -p tcp --dport 8080 -j ACCEPT
sudo iptables -t nat -A OUTPUT -o lo -p tcp --dport 80 -j REDIRECT --to-port 8080
sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080

Answer 3

On a quick glance, your setup seems to be correct.

• You have allowed INGRESS tcp:80 for all instances in the default network.
• Your VM is on the default network.

Traceroute will not give a good indication when you have VMs running on Cloud providers, because of the use of SDNs, virtual networks and whole bunch of intermediate networking infrastructure unfortunately.

One thing I notice is that your instance has 2 tags http-server and https-server. These could be used by some other firewall rules possibly which is somehow blocking traffic to your VM's tcp:80 port.

There are other variables in your setup and I'm happy to debug if needed further.

Tag based firewall rules

You can try tag based firewall rules which will apply the firewall rule only to instances which have the specified target tag.

Network tags are used by networks to identify which instances are subject to certain firewall rules and network routes. For example, if you have several VM instances that are serving a large website, tag these instances with a shared word or term and then use that tag to apply a firewall rule that allows HTTP access to those instances. Tags are also reflected in the metadata server, so you can use them for applications running on your instances. When you create a firewall rule, you can provide either sourceRanges or sourceTags but not both.

# Add a new tag based firewall rule to allow ingress tcp:80
gcloud compute firewall-rules create rule-allow-tcp-80 --source-ranges 0.0.0.0/0 --target-tags allow-tcp-80 --allow tcp:80

# Add the allow-tcp-80 target tag to the VM ssrf3
gcloud compute instances add-tags ssrf3 --tags allow-tcp-80

It might take a few seconds to couple of minutes for the changes to take effect.

NOTE: Since you're opening up ports of VM's external IPs to the internet, take care to restrict access accordingly as per the needs of your application running on these ports.