Bazel - Build, Push, Deploy Docker Containers to Kubernetes within Monorepo

Question

I have a monorepo with some backend (Node.js) and frontend (Angular) services. Currently my deployment process looks like this:

1. Check if tests pass
2. Build docker images for my services
3. Push docker images to container registry
4. Apply changes to Kubernetes cluster (GKE) with kubectl

I'm aiming to automate all those steps with the help of Bazel and Cloud Build. But I am really struggling to get started with Bazel:

To make it work I'll probably need to add a WORKSPACE file with my external dependencies and multiple BUILD files for my own packages/services? I need help with the actual implementation:

1. How to build my Dockerfiles with Bazel?
2. How push those images into a registry (preferably GCR)?
3. How to apply changes to Google Kubernetes Engine automatically?
4. How to integrate this toolchain with Google Cloud Build?

More information about the project

I've put together a tiny sample monorepo to showcase my use-case

Structure

├── kubernetes
├── packages
│   ├── enums
│   ├── utils
└── services
    ├── gateway

General

• Gateway service depends on enums and utils
• Everything is written in Typescript
• Every service/package is a Node module
• There is a Dockerfile inside the gateway folder, which I want to be built
• The Kubernetes configuration are located in the kubernetes folder.
• Note, that I don't want to publish any npm packages!

Answer 1

What we want is a portable Docker container that holds our Angular app along with its server and whatever machine image it requires, that we can bring up on any Cloud provider, We are going to create an entire pipeline to be incremental. "Docker Rules" are fast. Essentially, it provides instrumentality by adding new Docker layers, so that the changes you make to the app are the only things sent over the wire to the cloud host. In addition, since Docker images are tagged with a SHA, we only re-deploy images that changed. To manage our production deployment, we will use Kubernetes, for which Bazel rules also exist. Building a docker image from Dockerfile using Bazel is not possible to my knowledge because it's by design not allowed due to non-hermetic nature of Dockerfile. (Source: Building deterministic Docker images with Bazel)

The changes done as part of the source code are going to get deployed in the Kubernetes Cluster, This is one way to achieve the following using Bazel.

1. We have to put Bazel in watch mode, Deploy replace tells the Kubernetes cluster to update the deployed version of the app. a.

   Command : ibazel run :deploy.replace

2. In case there are any source code changes do it in the angular.

3. Bazel incrementally re-builds just the parts of the build graph that depend on the changed file, In this case, that includes the ng_module that was changed, the Angular app that includes that module, and the Docker nodejs_image that holds the server. As we have asked to update the deployment, after the build is complete it pushes the new Docker container to Google Container Registry and the Kubernetes Engine instance starts serving it. Bazel understands the build graph, it only re-builds what is changed.

Here are few Snippet level tips, which can actually help.

WORKSPACE FILE:

Create a Bazel Workspace File, The WORKSPACE file tells Bazel that this directory is a "workspace", which is like a project root. Things that are to be done inside the Bazel Workspace are listed below. • The name of the workspace should match the npm package where we publish, so that these imports also make sense when referencing the published package. • Mention all the rules in the Bazel Workspace using "http_archive" , As we are using the angular and node the rules should be mentioned for rxjs, angular,angular_material,io_bazel_rules_sass,angular-version,build_bazel_rules_typescript, build_bazel_rules_nodejs. • -Next we have to load the dependencies using "load". sass_repositories, ts_setup_workspace,angular_material_setup_workspace,ng_setup_workspace, • Load the docker base images also , in our case its "@io_bazel_rules_docker//nodejs:image.bzl", • Dont forget to mention the browser and web test repositaries web_test_repositories() browser_repositories( chromium = True, firefox = True, )

"BUILD.bazel" file.

• Load the Modules which was downloaded ng_module, the project module etc. • Set the Default visiblity using the "default_visibility" • if you have any Jasmine tests use the ts_config and mention the depndencies inside it. • ng_module (Assets,Sources and Depndeencies should be mentioned here ) • If you have Any Lazy Loading scripts mention it as part of the bundle • Mention the root directories in the web_package. • Finally Mention the data and the welcome page / default page.

Sample Snippet:

load("@angular//:index.bzl", "ng_module")
ng_module(
    name = "src",
    srcs = glob(["*.ts"]),
    tsconfig = ":tsconfig.json",
    deps = ["//src/hello-world"],
)
load("@build_bazel_rules_nodejs//:future.bzl", "rollup_bundle")
rollup_bundle(
  name = "bundle",
  deps = [":src"]
  entry_point = "angular_bazel_example/src/main.js"
)

Build the Bundle using the Below command.

bazel build :bundle

Pipeline : through Jenkins

Creating the pipeline through Jenkins and to run the pipeline there are stages. Each Stage does separate tasks, But in our case we use the stage to publish the image using the BaZel Run.

pipeline {
  agent any
  stages {
    stage('Publish image') {
      steps {
        sh 'bazel run //src/server:push'
      }
    }
  }
}

Note :

bazel run :dev.apply

1. Dev Apply maps to kubectl apply, which will create or replace an existing configuration.(For more information see the kubectl documentation.) This applies the resolved template, which includes republishing images. This action is intended to be the workhorse of fast-iteration development (rebuilding / republishing / redeploying).

2. If you want to pull containers using the workpsace file use the below tag

   container_pull( name = "debian_base", digest = "sha256:**", registry = "gcr.io", repository = "google-appengine/debian9", )

If GKE is used, the gcloud sdk needs to be installed and as we are using GKE(Google Contianer Enginer), It can be authenticated using the below method.

gcloud container clusters get-credentials <CLUSTER NAME>

The Deploymnet Object should be mentioned in the below format:

load("@io_bazel_rules_k8s//k8s:object.bzl", "k8s_object")

k8s_object(
  name = "dev",
  kind = "deployment",
  template = ":deployment.yaml",
  images = {
    "gcr.io/rules_k8s/server:dev": "//server:image"
  },
)

Sources :

• https://docs.bazel.build/versions/0.19.1/be/workspace.html
• https://github.com/thelgevold/angular-bazel-example
• https://medium.com/@Jakeherringbone/deploying-an-angular-app-to-kubernetes-using-bazel-preview-91432b8690b5
• https://github.com/bazelbuild/rules_docker
• https://github.com/GoogleCloudPlatform/gke-bazel-demo
• https://github.com/bazelbuild/rules_k8s#update
• https://codefresh.io/howtos/local-k8s-draft-skaffold-garden/
• https://github.com/bazelbuild/rules_k8s

Answer 2

A few months later and I've gone relatively far in the whole process. Posting every detail here would just be too much!

So here is the open-source project which has most of the requirements implemented: https://github.com/flolu/fullstack-bazel

Feel free to contact me with specific questions! :)

Good luck