How can Java build tools stop rebuilding everything?

Question

• tl;dr Version: Is there a way I can tell ant/maven/gradle to track file dependencies and only rebuild files which are outdated, i.e., older than their dependencies?

• long version below:

How I did it in C/C++

I used make for many years for C/C++. And while it might be cumbersome sometimes, it at least does one of the most important things for me: It tracks file dependencies and only rebuilds a file if any of its dependencies is newer than the file itself. This is great, it reduces build times considerably especially when a lot of code generation tools are executed.

I use a lot of code generation in my projects. However, code generation takes some time and the generated code has to be compiled afterwards, so even more time wasted. Most of the generated code depends only on a small number of files, e.g., a grammar specification when generating a parser. With make, I simply set dependencies to the specification file and the parser is only rebuilt when the spec changes. In some of my projects generating the code takes 80-90% of the time a complete rebuild requires. Thus, not having to rebuild generated code speeds up my build process by almost a factor of 10.

Now to java

People don't use make for java, so even if I like make, I also the usual java build tools for my java projects. It seems that the build tools I tried (ant,gradle,maven) do not offer such dependency tracking. Instead, they rely on the called applications or tasks to do this (e.g., the ant copy task copies only files which are outdated). However, the code generation programs I use simply do not offer such checks. They always execute the full code generation no matter whether the input specifications are outdated or not.

Consequently, my build times in java got longer and longer. They got so long that I took the code generation out of the usual build cycle into a separate target and only executed this target manually after changing any spec file. But of course, this is error prone since I sometimes forget it. My build process now feels totally flawed.

Finally, my questions

1. Is there a way to perform dependency tracking and rebuilding only outdated files in any of the build tools I mentioned (ant,gradle,maven) that I am not aware of?

2. If not, then why? Why does the java community not seem to care about this feature? Especially when a lot of code generation is involved, such a feature is extremely important, especially if the code generation tools used do not offer that feature themselves.

3. If not, then are there any other java build tools which do have such tracking?

Answer 1

Some Gradle-centric answers:

Gradle will only rerun a task if its inputs have changed compared to the previous run, or its outputs have been deleted or modified. This is a generic feature that works for any task as long as it declares its inputs and outputs. (Built-in task types declare their inputs and outputs out-of-the-box.) This feature is often called incremental build - a build that only executes a subset of tasks.

Gradle also has (very) experimental support for incremental Java compilation. This is an example for an incremental task, i.e. a task that only does a subset of work depending on which inputs have changed compared to the previous run. Implementing an incremental task requires knowledge of the task's domain (e.g. Java compilation). As such this feature is less generic than the incremental build feature and requires more effort from the task implementor.

To give a concrete example, if your code generation task declares its inputs and outputs (which is very easy to do), it will only be rerun if a grammar file has changed or the generated files have been deleted or modified. (For the purpose of this example, let's assume there can be multiple grammar files.) Additionally, your task implementation could ask Gradle which grammar files have changed since the last run, and use this information to only regenerate a subset of files. Obviously this would take effort to implement on your side, and may not be possible to achieve depending on the nature of the underlying tools used by your task.

Some more words on incremental compilation: Incremental compilation is less widespread among Java build tools for several reasons. For one thing, Java compiles much faster than C++. Also, code bases are typically modularized into many smaller compilation units, each of which will only get recompiled if one of its inputs has changed (at least in Gradle). Nevertheless, I'm sure that more Java build tools will support incremental compilation over time. Some already do today (e.g. Pants, sbt, and IDEs such as Eclipse and IntelliJ), and Gradle is about to catch up.