How do I use the file provisioner with google_compute_instance_template?

Question

Using Terraform, I need to copy files to Google Compute Engine instance templates. For this I normally use the file provisioner, but it doesn't work since it depends on SSH connections, which fail due to requiring externally accessible host addresses. Due to the dynamic nature of instance templates, I don't know how to assign externally accessible host addresses to instances.

How can I implement copying of files to GCE instances created through instance templates (via Terraform)?

Example Terraform google_compute_instance_template Definition

resource "google_compute_instance_template" "node" {
  name = "kubernetes-node-template"
  machine_type = "g1-small"
  can_ip_forward = true
  tags = ["staging", "node"]

  network_interface {
    network = "default"
  }

  provisioner "file" {
    source = "worker/assets/kubelet.service"
    destination = "/etc/systemd/system/kubelet.service"
  }

  connection {
    user = "core"
    type = "ssh"
    private_key = "${file("~/.ssh/id_rsa")}"
  }
}

Answer 1

I was able to resolve this issue with the following configuration.

resource "google_compute_instance" "hubmud" {
    name = "hubmud"
    machine_type = "f1-micro"

    tags = ["buildserver", "jenkins", "central", "terraformer"]
    tags = [ "http-server" ]

    zone = "us-central1-b"

    disk {
        image = "ubuntu-1404-trusty-v20160406"
    }

    network_interface {
        network = "default"
        access_config {}
    }

    provisioner "file" {
        source = "installations.sh"
        destination = "installations.sh"
        connection {
            type = "ssh"
            user = "ubuntu"
            private_key = "${file("~/.ssh/google_compute_engine")}"
        }
    }

    provisioner "remote-exec" {
        inline = [
          "chmod +x ~/installations.sh",
          "cd ~",
          "./installations.sh"
        ]
        connection {
            type = "ssh"
            user = "ubuntu"
            private_key = "${file("~/.ssh/google_compute_engine")}"
        }

    }

    service_account {
        scopes = ["userinfo-email", "compute-ro", "storage-ro"]
    }
}

The SSH key I used was the one generated by the gcloud instance, and the file to copy has to be in the format as shown. I did run into problems with using ~/ or just ./ to designate where the file is. It's also important to note that the file copied under the "ubuntu" account, which appears to be a default account on ubuntu that GCE has on the image by default.

Just to note, I also added the type = "ssh" even though that is the default for the connection type, so it isn't needed. I like to be verbose with some things in my configuration files though, so I added it.

Answer 2

you can SSH into google_compute_instances (this is a concrete, tangible VM afterall), but not into neither a google_compute_instance_group_manager nor into a google_compute_instance_template as these are abstractions.

as stated in here https://github.com/terraform-providers/terraform-provider-google/issues/52, this is not possible at the moment.

Answer 3

Perhaps this feature was added later, but as of early 2021 you can use the optional metadata_startup_script argument of the google_compute_instance_template to specify a script that will run each time any instance created from the template starts up:

resource "google_compute_instance" "hubmud" {
    name = "hubmud"
    machine_type = "f1-micro"

    tags = ["buildserver", "jenkins", "central", "terraformer"]
    tags = [ "http-server" ]

    zone = "us-central1-b"

    disk {
        image = "ubuntu-1404-trusty-v20160406"
    }

    network_interface {
        network = "default"
        access_config {}
    }

    # As per the documentation for google_compute_instance_template (https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/compute_instance_template#metadata_startup_script)
    # you can set either metadata.startup-script or metadata_startup_script
    # to the contents of a file that will be added to the metadata of
    # any instance created from the template. 
    # In both cases the script will run each time the instance starts. 
    # The difference is that if metadata_startup_script is used and the metadata 
    # value is changed the instance will be destroyed and recreated before 
    # running the new script.
    metadata_startup_script = file("${path.module}/installations.sh")

    service_account {
        scopes = ["userinfo-email", "compute-ro", "storage-ro"]
    }
}

Note that this mechanism requires OS support, but as the documentation for the underlying arguments on google_compute_instance points out:

"Most linux-based images will run the content of metadata.startup-script in a shell on every boot. At a minimum, Debian, CentOS, RHEL, SLES, Container-Optimized OS, and Ubuntu images support this key. Windows instances require other keys depending on the format of the script and the time you would like it to run."

Tested as far back as Ubuntu 16.04, which reaches EOL for LTS at the end of this month. So this feature has been around for 5+ years. Not clear if it was exposed by the GCP Terraform provider that long ago, but it works perfectly now.