How to Bind and Send from Google Cloud Forwarding Rule IP Address?

Question

I've followed the instructions for Using Protocol Forwarding on the Google Cloud Platform. So I now have something like this:

$ gcloud compute forwarding-rules list
NAME    REGION    IP_ADDRESS      IP_PROTOCOL  TARGET
x-fr-1  us-west1  104.198.?.??    TCP          us-west1-a/targetInstances/x-target-instance
x-fr-2  us-west1  104.198.?.??    TCP          us-west1-a/targetInstances/x-target-instance
x-fr-3  us-west1  104.198.??.???  TCP          us-west1-a/targetInstances/x-target-instance
x-fr-4  us-west1  104.198.??.???  TCP          us-west1-a/targetInstances/x-target-instance
x-fr-5  us-west1  104.198.?.???   TCP          us-west1-a/targetInstances/x-target-instance

(Note: Names have been changed and question-marks have been substituted. I'm not sure it matters to keep these private but better safe than sorry.)

My instance "x" is in the "x-target-instance" and has five forwarding rules "x-fr-1" through "x-fr-5". I'm running nginx on "x" and I can access it from any of its 6 external IP addresses (1 for the instance + 5 forwarding rules). So far, so good.

I am interested now in binding a server to these external IP addresses. To explore, I tried using Python:

import socket
import time

def serve(ip_address, port=80):
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.bind((ip_address, port))
    try:
        sock.listen(5)
        while True:
            con, _ = sock.accept()
            print con.getpeername(), con.getsockname()
            con.send(time.ctime())
            con.close()
    finally:
        sock.close()

Now I can bind "0.0.0.0" and I get some interesting results:

>>> serve("0.0.0.0")
('173.228.???.??', 57288) ('10.240.?.?', 80)
('173.228.???.??', 57286) ('104.198.?.??', 80)

When I communicate with the server on its external IP address, the "getsockname" method returns the instance's internal IP address. But when I communicate with the server on an external IP address as used by a forwarding rule, then the "getsockname" methods returns the external IP address.

Ok, now I bind the instance's internal IP address:

>>> serve("10.240.?.?")
('173.228.???.??', 57295) ('10.240.?.?', 80)

Again I can communicate with the server on its external IP address, and the "getsockname" method returns the instance's internal IP address. That seems a bit odd.

Also, if I try to bind the instance's external IP address:

>>> serve("104.198.?.??")
error: [Errno 99] Cannot assign requested address

Then I get an error.

But, if I try to bind the external IP addresses used by the forwarding rules and then make a request:

>>> serve("104.198.??.???")
('173.228.???.??', 57313) ('104.198.??.???', 80)

It works.

Finally I look at "ifconfig":

ens4      Link encap:Ethernet  HWaddr 42:01:0a:??:??:??  
          inet addr:10.240.?.?  Bcast:10.240.?.?  Mask:255.255.255.255
          inet6 addr: fe80::4001:???:????:2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1460  Metric:1
          RX packets:37554 errors:0 dropped:0 overruns:0 frame:0
          TX packets:32286 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:41201244 (41.2 MB)  TX bytes:3339072 (3.3 MB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:9403 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9403 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:3155046 (3.1 MB)  TX bytes:3155046 (3.1 MB)

And I see only two interfaces. Clearly, the abilities of Google Cloud Platform Networking has exceeded what I can remember from my Computer Networking class in college. To summarize my observations:

1. If I want to bind on the instance's external IP address, then I bind its internal IP address.
2. A process bound to the instance's internal IP address can not differentiate the destination IP between the instance's internal or external IP addresses.
3. The single networking adapter, "ens4", is receiving packets bound for any of the instance's 6 external IP address.

And here's my questions:

1. Why can I not bind the instance's external IP address?
2. How is it that I can bind the external IP addresses used by forwarding rules when I have no associated network adapters?
3. If I want to restrict SSH access to the instance's external IP address, should I configure SSH to bind the internal IP address?
4. If I setup an HTTP proxy on one of the external IP addresses used by a forwarding rule, what will be the source IP of the proxied request?
5. Lastly, and this may be a bug, why is the forwarding rules list empty in the web interface at https://console.cloud.google.com/networking/loadbalancing/advanced/forwardingRules/list?project=xxx when I can see them with "gcloud compute forwarding-rules list"?

Answer 1

1. it's not in the local routing table ('ip route show table local') [ you could of course add it (e.g. 'ip address add x.x.x.x/32 dev ens4'), but doing so wouldn't do you much good, since no packets will be delivered to your VM using that as the destination address - see below... ]
2. because the forwarded addresses have been added to your local routing table ('ip route show table local')
3. you could [ though note that this would restrict ssh access to either external clients targeting the external IP address, or to clients within your virtual network targeting either the external or internal IP address ]. However, as already noted - it might be more important to restrict the allowed client addresses (not the server address), and for that the firewall would be more effective.
4. it depends on where the destination of the proxied request goes. If it's internal to your virtual network, then it will be the VM's internal IP address, otherwise it's NAT-ed (outside of your VM) to be the VM's external IP address.
5. There are multiple tabs on that page - two of which list different classes of forwarding rule ("global forwarding rules" vs "forwarding rules"). Admittedly somewhat confusing :P

One other thing that's slightly confusing - when sending packets to your VM using its external IP as the destination address, an entity outside the VM (think of it as a virtual switch / router / NAT device) automatically modifies the destination to be the internal IP before the packet arrives at the virtio driver for the virtual NIC - so there's nothing you can do to modify that behavior. Packets addressed to the IP of a forwarding rule, however, are not NAT-ed (as you've seen).

Hope that helps!