Bazel: genrule that outputs a directory

Question

I'm just getting started working with Bazel. So, I apologize in advance that I haven't been able to figure this out.

I'm trying to run a command that outputs a bunch of files to a directory and make this directory available for subsequent targets. I have two different attempts:

1. Use genrule
2. Write my own rule

I was naively hoping to just do this with a genrule. But, it doesn't seem you can say "I don't know exactly what this command is going to output" and put a directory in outs. Now I'm trying to write a rule that can use ctx.actions.declare_directory but I haven't gotten it quite right. I can't seem to get the tools over from my workspace and into my rule.

My genrule attempt looks something like this:

genrule(
    name = "doit",
    srcs = [
        "doitConfigA",
        "doitConfigB",
    ],
    cmd = 'HOME=. ./$(location path/to/doit) install',

    # Neither of the below outs work - seems like bazel wants to know
    # exactly this list of files. I don't know the files that
    # will be output ahead of time.

    # This one looks at the `out_dir` that I already have and
    # expects the files to be the same which they might not be
    outs = glob(["out_dir/**/*.*"]),

    # this fails with:
    # "declared output 'out_dir' was not 
    # created by genrule. This is probably because the genrule actually 
    # didn't create this output, or because the output was a directory 
    # and the genrule was run remotely (note that only the contents of 
    # declared file outputs are copied from genrules run remotely)"
    outs = ['out_dir'],
    tools = ['path/to/doit'],
)

My custom rule attempt looks something like this:

def _impl(ctx):
  dir = ctx.actions.declare_directory("out_dir")

  ctx.actions.run_shell(
      outputs=[dir],
      progress_message="Running doit install ...",
      command="HOME=. ./path/to/doit install",
      tools=[ctx.attr.tools],
  )

doit = rule(
    implementation=_impl,
    attrs={
      "tools": attr.label_list(allow_files=True),
    },
    outputs={"out": "out_dir"},
)

Then, to run my doit rule, my BUILD file looks like this:

doit(
  name = 'doit',
  tools = ['path/to/doit'],
)

In my genrule, the command runs but it doesn't like my trying to use a directory in outs, it seems. In my custom rule, I can't seem to tell Bazel that I want to use ./path/to/doit as a tool from my workspace, eg expected type 'File' for 'tools' element but got type 'list' instead ...

Seems like I must be missing something basic because surely this is a common situation to run a command and output a bunch of unknown stuff to a directory?

Answer 1

The output of a genrule must be a fixed list of files. As a work-around, you can create a zip from the output directory.

I used this approach to manipulate the output of yarn install where the usual method was not viable:

genrule(
  name = "node_modules",
  srcs = [
    "package.json",
    "yarn.lock",
  ],
  cmd = " && ".join([
    "yarn install --pure-lockfile",
    "zip -r $@ node_modules",
  ]),
  outs = [
    "node_modules.zip",
  ],
)

Then a rule that consumes the zip:

# Rule that generates a list of the folders in node_modules
genrule(
  name = "node_modules_ls",
  srcs = [
    ":node_modules",
  ],
  cmd = " && ".join([
    "unzip $(location :node_modules) -d . ",
    "ls > $@",
  ]),
  outs = [
    "out.txt",
  ],
)

Answer 2

A while ago I created this example showing how to use directories with skylark action: How to build static library from the Generated source files using Bazel Build. Maybe it still works :)

Genrule won't work, this is too advanced use case.

Answer 3

https://github.com/aspect-build/bazel-lib/blob/main/docs/run_binary.md has a similar API to genrule, and it supports directory outputs.