Verify step in bazel

Question

I'm looking for a good recipe to run "checks" or "verify" steps in Bazel, like go vet, gofmt, pylint, cppcheck. These steps don't create any output file. The only thing that matters is the return code (like a test).

Right now I'm using the following recipe:

sh_test(
    name = "verify-pylint",
    srcs = ["verify-pylint.sh"],
    data = ["//:all-srcs"],
)

And verify-pylint.sh looks like this:

find . -name '*.py' | xargs pylint

This has two problems:

• The verify logic is split between the shell script and the BUILD file. Ideally I would like to have both in the same place (in the BUILD file)
• Anytime one of the source file changes (in //:all-srcs), bazel test verify-pylint re-runs pylint on every single file (and that can be expensive/slow).

What is the idiomatic way in bazel to run these steps?

Answer 1

There are more than one solutions.

The cleanest way is to do the verification at build time: you create a genrule for each file (or batch of files) you want to verify, and if verification succeeds, the genrule outputs something, if it fails, then the rule outputs nothing, which automatically fails the build as well.

Since success of verification depends on the file's contents, and the same input should yield the same output, the genrules should produce an output file that's dependent on the contents of the input(s). The most convenient thing is to write the digest of the file(s) to the output if verification succeeded, and no output if verification fails.

To make the verifier reusable, you could create a Skylark macro and use it in all your packages.

To put this all together, you'd write something like the following.

Contents of //tools:py_verify_test.bzl:

def py_verify_test(name, srcs, visibility = None):
    rules = {"%s-file%d" % (name, hash(s)): s for s in srcs}
    for rulename, src in rules.items():
        native.genrule(
            name = rulename,
            srcs = [s],
            outs = ["%s.md5" % rulename],
            cmd = "$(location //tools:py_verifier) $< && md5sum $< > $@",
            tools = ["//tools:py_verifier"],
            visibility = ["//visibility:private"],
        )

    native.sh_test(
        name = name,
        srcs = ["//tools:build_test.sh"],
        data = rules.keys(),
        visibility = visibility,
    )

Contents of //tools:build_test.sh:

#!/bin/true
# If the test rule's dependencies could be built,
# then all files were successfully verified at
# build time, so this test can merely return true.

Contents of //tools:BUILD:

# I just use sh_binary as an example, this could
# be a more complicated rule of course.
sh_binary(
    name = "py_verifier",
    srcs = ["py_verifier.sh"],
    visibility = ["//visibility:public"],
)

Contents of any package that wants to verify files:

load("//tools:py_verify_test.bzl", "py_verify_test")

py_verify_test(
    name = "verify",
    srcs = glob(["**/*.py"]),
)

Answer 2

A simple solution.

In your BUILD file:

load(":gofmt.bzl", "gofmt_test")

gofmt_test(
    name = "format_test",
    srcs = glob(["*.go"]),
)

In gofmt.bzl:

def gofmt_test(name, srcs):
  cmd = """
    export TMPDIR=.
    out=$$(gofmt -d $(SRCS))

    if [ -n "$$out" ]; then
      echo "gmfmt failed:"
      echo "$$out"
      exit 1
    fi
    touch $@
  """
  native.genrule(
      name = name,
      cmd = cmd,
      srcs = srcs,
      outs = [name + ".out"],
      tools = ["gofmt.sh"],
  )

Some remarks:

• If your wrapper script grows, you should put it in a separate .sh file.
• In the genrule command, we need $$ instead $ due to escaping (see documentation)
• gofmt_test is actually not a test and will run with bazel build :all. If you really need a test, see Laszlo's example and call sh_test.
• I call touch to create a file because genrule requires an output to succeed.
• export TMPDIR=. is needed because by default the sandbox prevents writing in other directories.

To cache results for each file (and avoid rechecking a file that hasn't changed), you'll need to create multiple actions. See Laszlo's for loop.

To simplify the code, we could provide a generic rule. Maybe this is something we should put in a standard library.