How can I use the Shake library to build a reactive build system? [closed]

Question

Would it be possible to make Shake reactive, using inotify (or whatever git-annex and Yesod use) so that if ever the filesystem changes in such a way to imply that rule should execute, it does so at the earliest opportunity?

Answer 1

The author of Shake, Neil Mitchell, answered this by saying:

There are a few ways to approach this:

1. You could just rerun Shake every time it detects something has changed. Shake is highly optimised for fast rebuilds, if the change requires doing a compile, then the time for Shake to figure out what to rebuild is probably minimal. Requires no changes to Shake.

2. There are certain things Shake does on startup, like reading the Shake database. If there is demand, and that turns out to be noticeable in time, I would happily provide a rerun Shake cheaply API of some sort - it's not that difficult to do.

3. When Shake does do a rebuild-check, the most expensive thing it does is checking file modification times. If the inotify layer gave a list of the files that had changed I could only recheck things that had actually changed. For a massive project you're likely to see ~1s checking modification times, so it probably buys you a little, and isn't too hard to implement.

4. If Shake is actively building, and then something changes, you could throw an exception, kill whatever is being built, and restart Shake. Shake has been thoroughly tested with exceptions being thrown at it, and does the right thing. I know at least one person uses Shake in this way.

5. Finally, if Shake is actively building, you could dynamically terminate just those rules whose inputs have changed and go again. Shake could support this model, but it would be a reasonable amount of work, and require re-engineering some pieces. That would be the full reactive model, but I suspect it only starts to be a benefit when you have a massive number of files and a few files are changing almost continuously but most files aren't.

We also determined that combining Shake with a utility like Hobbes (also on Hackage) can make it possible to do reactive builds.