Can Google Cloud Dataflow be run without an external IP address?

Question

Something we noticed while working on scaling up our usage of Google Cloud Dataflow was that we would start to run up against our quota on In-use IP addresses. We are aware that requesting a quota increase is an option (and have already done so and been approved as such for CPU's) but wanted to know if it is (or is going to be) possible to run Dataflow instances without external IP addresses (both for quota reasons and because of the cost of IP addresses).

Answer 1

It's possible now. From the documentation:

To turn off public IPs:

Enable Private Google Access for your network or subnetwork.

In the parameters of your Cloud Dataflow job, specify --usePublicIps=false and --network=[NETWORK] or --subnetwork=[SUBNETWORK].

More on the option usePublicIps

usePublicIps

boolean

Specifies whether Cloud Dataflow workers use public IP addresses. If the value is set to false, Cloud Dataflow workers use private IP addresses for all communication. In this case, if the subnetwork option is specified, the network option is ignored. Make sure that the specified network or subnetwork has Private Google Access enabled. If not set, the default value is true and Cloud Dataflow workers use public IP addresses.