GitHub Cloud Build Integration with multiple cloudbuild.yamls in monorepo

Question

GitHub's Google Cloud Build integration does not detect a cloudbuild.yaml or Dockerfile if it is not in the root of the repository.

When using a monorepo that contains multiple cloudbuild.yamls, how can GitHub's Google Cloud Build integration be configured to detect the correct cloudbuild.yaml?

File paths:

services/api/cloudbuild.yaml
services/nginx/cloudbuild.yaml
services/websocket/cloudbuild.yaml

Cloud Build integration output:

Answer 1

You can do this by adding a cloudbuild.yaml in the root of your repository with a single gcr.io/cloud-builders/gcloud step. This step should:

1. Traverse each subdirectory or use find to locate additional cloudbuild.yaml files.
2. For each found cloudbuild.yaml, fork and submit a build by running gcloud builds submit.
3. Wait for all the forked gcloud commands to complete.

There's a good example of one way to do this in the root cloudbuild.yaml within the GoogleCloudPlatform/cloud-builders-community repo.

If we strip out the non-essential parts, basically you have something like this:

steps:
- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    for d in */; do
      config="${d}cloudbuild.yaml"
      if [[ ! -f "${config}" ]]; then
        continue
      fi

      echo "Building $d ... "
      (
        gcloud builds submit $d --config=${config}
      ) &
    done
    wait

Answer 2

We are migrating to a mono-repo right now, and I haven't found any CI/CD solution that handles this well.

The key is to not only detect changes, but also any services that depend on that change. Here is what we are doing:

• Requiring every service to have a MAKEFILE with a build command.
• Putting a cloudbuild.yaml at the root of the mono repo
• We then run a custom build step with this little tool (old but still seems to work) https://github.com/jharlap/affected which lists out all packages have changed and all packages that depend on those packages, etc.
• then the shell script will run make build on any service that is affected by the change.

So far it is working well, but I totally understand if this doesn't fit your workflow.

Another option many people use is Bazel. Not the most simple tool, but especially great if you have many different languages or build processes across your mono repo.