Setup Cloud NAT for public GKE clusters

Question

I'd like to setup a NAT gateway, using Cloud NAT, so that VMs/Pods in a public GKE cluster use static IP addresses.

The issue I'm facing is that the NAT gateway seems to only be used if VMs have no other options, i.e:

GCP forwards traffic using Cloud NAT only when there are no other matching routes or paths for the traffic.

But in the case of a public GKE cluster, VMs have ephemeral external IPs, so they don't use the gateway.

According to the doc:

If you configure an external IP on a VM's interface [...] NAT will not be performed on such packets. However, alias IP ranges assigned to the interface can still use NAT because they cannot use the external IP to reach the Internet.

And

With this configuration, you can connect directly to a GKE VM via SSH, and yet have the GKE pods/containers use Cloud NAT to reach the Internet.

That's what I want, but I fail to see what precisely to setup here.

What is implied by alias IP ranges assigned to the interface can still use NAT and how to set this up?

Answer 1

Using google's Cloud NAT with public GKE clusters works!

First a cloud NAT gateway and router needs to be setup using a reserved external IP.

Once that's done the ip-masq-agent configuration needs to be changed to not masquerade the pod IPs for the external IPs that are the target of requests from inside the cluster. Changing the configuration is done in the nonMasqueradeCidrs list in the ConfigMap for the ip-masq-agent.

The way this works is that for every outgoing requests to an IP in the nonMasqueradeCidrs list IP masquerading is not done. So the requests does not seem to originate from the node IP but from the pod IP. This internal IP is then automatically NATed by the Cloud NAT gateway/router. The result is that the request seems to originate from the (stable) IP of the cloud NAT router.

Sources:

• https://rajathithanrajasekar.medium.com/google-cloud-public-gke-clusters-egress-traffic-via-cloud-nat-for-ip-whitelisting-7fdc5656284a
• https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent

Answer 2

The idea here is that if you use native VPC (Ip alias) for the cluster, your pods will not use SNAT when routing out of the cluster. With no SNAT, the pods will not use the node's external IP and thus should use the Cloud NAT.

Unfortunately, this is not currently the case. While Cloud NAT is still in Beta, certain settings are not fully in place and thus the pods are still using SNAT even with IP aliasing. Because of the SNAT to the node's IP, the pods will not use Cloud NAT.

This being said, why not use a private cluster? It's more secure and will work with Cloud NAT. You can't SSH directly into a node, but A) you can create a bastion VM instance in your project that can SSH using the internal IP flag and B) you generally do not need to SSH into the node on most occassions.