Bigquery Terraform Schedule queries deployment error

Question

Below is code from my Bigquery terraform main.tf file

    resource "google_bigquery_data_transfer_config" "config_queries" {
  for_each = fileset("${path.module}/scheduled_queries", "*.sql")
  depends_on = [google_bigquery_table.tables]
  data_source_id = "scheduled_queries"
  location=var.location
  destination_dataset_id=google_bigquery_dataset.dataset.dataset_id
  display_name = "config_queries_${substr(each.value, 0, length(each.value) -4)}"
  schedule = "every 4 hours "
 params = {
    destination_table_name_template = substr(each.value,0,length(each.value) -4)
    write_disposition = "WRITE_APPEND"
    query = file("${path.module}/scheduled_queries/${each.value}")
  }
}

I am getting below error:

│ Error: Error creating Config: googleapi: Error 404: Requested entity was not found.

The same code is working fine if running sql files individually. Any suggestions

Answer 1

I had the same error, but while creating a single scheduled query. It turned out, the problem was me taking the liberty of changing data_source_id value into some custom string. After reverting to "scheduled_query" string, as it is provided in available examples, it worked. I believe the solution provided above deals with a different problem (IAM), which can be somewhat simplified by using explicit service account with BigQuery Admin role applied (service_account_name TF argument), after enabling bigquerydatatransfer API.