Significant delay when accessing internet via google cloud NAT [closed]

Question

in our setup we have VM without external IPs accessing outside world via Google Cloud NAT.

We are observing strange behavior:

• every 10 - 15th request is significantly delayed (15 - 135 seconds)

  ~$ for i in {1..50}; do echo $i;  time curl -s 'https://stackoverflow.com/' > 
  /dev/null; done
  .
  .
  .
  15

  real  2m10.830s
  user  0m0.049s
  sys   0m0.029s

• I can see in the stackdriver logs that concerned connections were DROPPED:

    {
    .
    .
    .
      jsonPayload: {
        allocation_status: "DROPPED"
      }
      timestamp: "2019-09-20T11:41:05.904608023Z"  
    }

Any ideas/suggestions how to resolve the issue will be highly appreciated. :)

Thanks,

Boris

Answer 1

It is possible your VMs are running out of available NAT ports. For example, if you allocate 4096 ports per VM, then you need a NAT IP address for every 15 VMs. During live operation if the VMs + NAT run out of available ports due to open connections (TCP being the longest lived) you may begin to see dropped traffic. You may want to consider reducing the TCP timeout, or allocating more ports per VM.

You can see the current state of the NAT mapping (for example, number of ports per IP/VM) by using this command in the Cloud console:

gcloud compute routers get-nat-mapping-info [ROUTER NAME]