Guidelines for splitting up a very large Gradle project with lots of sub projects to make Gradle Build Faster and IntelliJ Gradle Refresh faster

Question

I have a pretty large git project that I have a large single build.gradle that defines the sub projects.. For the first few years... no problem. After the first few years it gets kind of slow on the Gradle Command line configuration step because of the number of sub projects.

For example, when I run a gradle command I see a pause on: > Configuring > 0/77 projects > ...

The pause gets bigger with each project added it seems like.

Another big problem is when i go to import the project or refresh the gradle deps with intelliJ, it takes a long time on the "analyzing dependencies" really delaying development as we need to have a pretty large wait everytime we sync dependencies.

What do people do in the field typically to get passed this? I have had to just eat the delay which is painful.

Do people usually split up the build.gradle file into unrelated chunks so that you get a network of multi-project builds? Or should there be a way to make that not be so damn slow?

• I have analyzed the dependencies, all pretty simple actually.
• Ondemand, Daemon and, parallel deploys happen in some limited way.

But the same problem stands...

What other tricks are out there? My plan is to separate the main parts of my code in sections and load them independently so that the gradle commands and intelliJ refreshes are smaller UP-TO-DATE check and configuration setup. It would be nice if we didn't have to split up our build.gradle file just because of sub-project limit.

Answer 1

Following our discussion in the comments of your question, it is fairly clear that you don't want to go down the route of publishing some of the interface jars to a repository to remove some of the subprojects.

Given that, here are a few suggestions. I know you have already tried some of them, but there are a few here that you haven't mentioned above which might help you slightly:

Profiling and dry run

First of all, try using the following command:

    ./gradlew {task} --dry-run --profile

This will perform a dry run of the task with name {task} and create a profile html file under $buildDir/reports/profile. You can use this to see which tasks are taking up the majority of the time across congifuration, dependency resolution and task execution. If you can see that one task is taking a particularly long time, have a look at it and see if you can cut down what is happening in that task.

Configuration on demand

To reduce configuration time, try enabling configuration on demand in your gradle.properties file like so:

    org.gradle.configureondemand=true

This will mean that instead of configuring everything, gradle will attempt to only configure the necessary projects required to run a task.

Use the gradle daemon

The gradle daemon is a background process which doesn't exit when your gradle task finishes, and can be reused. This limits the amount of overhead of starting a new gradle jvm process every build. To activate the gradle daemon, add the following line to your gradle.properties file:

    org.gradle.daemon=true

Use the most recent possible versions of Gradle and Java

Something is simple as upgrading your gradle version can improve speed. If you haven't upgraded for a while, or are using a fairly old version, consider upgrading. There may be some initial pain in upgrading, but it will be worth it in the long run. The same applies for upgrading java versions. Obviously, this is not always possible due to other constraints, but try and use the most recent java version available to you.

Avoid using dynamic dependencies

Gradle allows your to specify ranges for dependencies that your project is able to use like so:

dependencies {
    compile "commons-lang:commons-lang:2.+"
}

This means that gradle will always try and find the latest version of the dependency that matches the constraint. This flexibility comes at a performance cost as gradle will have to go online to check what version to use. For this reason, and a few others that I won't go into here, it is a good idea to specify the version of a dependency explicitly.

Parallelize the build

This allows unrelated subprojects to be built in parallel, which speeds up the build process. To enable this, set the following in gradle.properties:

org.gradle.parallel=true

Ignore non-relevant tasks

Say you want to do a build, but don't necessarily want to run all of the tests again. You can run the following task, and the test task will not be run. This can significantly increase build time:

./gradlew build -x test

EDIT -- Following some further research, and after finding this answer, I had further suggestions.

Disable parts of the build using properties

As shown in the linked question, if you have a project and you only want to run the functional tests under certain circumstances, you can add the following to the root build.gradle file:

project('functional-tests') {
test {
    onlyIf {
        project.hasProperty("functionalTests")
    }
}

}

This means that functional tests will only be run if you run a task like so:

./gradlew test -PfunctionalTests

Disable certain subprojects in the settings.gradle

You could also disable certain subprojects in the settings.gradle like so:

if (someBoolean) {
    include 'functional-tests'
}