Can a shake rule determine which "needs" have changed since the last build?

Question

I am building a shake based build system for a large Ruby (+ other things) code base, but I am struggling to deal with Ruby commands that expect to be passed a list of files to "build".

Take Rubocop (a linting tool). I can see three options:

• need all Ruby files individually; if they change, run rubocop against the individual file that changed for each file that changed (very slow on first build or if many ruby files change because rubocop has a large start up time)
• need all Ruby files; if any change, run rubocop against all the ruby files (very slow if only one or two files have changed because rubocop is slow to work out if a file has changed or not)
• need all Ruby files; if any change, pass rubocop the list of changed dependencies as detected by Shake

The first two rules are trivial to build in shake, but my problem is I cannot work out how to represent this last case as a shake rule. Can anyone help?

Answer 1

There are two approaches to take with Shake, using batch or needHasChanged. For your situation I'm assuming rubocop just errors out if there are lint violations, so a standard one-at-a-time rule would be:

"*.rb-lint" %> \out -> do
    need [out -<.> "rb"]
    cmd_ "rubocop" (out -<.> "rb")
    writeFile' out ""

Use batch

The function batch describes itself as:

Useful when a command has a high startup cost - e.g. apt-get install foo bar baz is a lot cheaper than three separate calls to apt-get install.

And the code would be roughly:

batch 3 ("*.rb-lint-errors" %>)
    (\out -> do need [out -<.> "rb"]; return out) $
    (\outs -> do cmd_ "rubocop" [out -<.> "rb" | out <- outs]
                 mapM_ (flip writeFile' "") pits)

Use needHasChanged

The function needHasChanged describes itself as:

Like need but returns a list of rebuilt dependencies since the calling rule last built successfully.

So you would write:

 "stamp.lint" *> \out -> do
     changed <- needHasChanged listOfAllRubyFiles
     cmd_ "rubocop" changed
     writeFile' out ""

Comparison

The advantage of batch is that it is able to run multiple batches in parallel, and you can set a cap on how much to batch. In contrast needHasChanged is simpler but is very operational. For many problems, both are reasonable solutions. Both these functions are relatively recent additions to Shake, so make sure you are using 0.17.2 or later, to ensure it has all the necessary bug fixes.