SBT build structure for a mono repository

Question

I'd like to use SBT for a build structured around a single Git repository with tens of projects. I'd like to have the following possibilities from the build:

1. One-click build/test/release of all projects in the build.
2. Define and apply common settings for all the projects in the build.
3. Define project-specific settings in the project sub-directories, keeping the root of the build clean.
4. Project should be able depend on each other, but...
5. Most of the projects will not depend on each other in the classpath sense.
6. Some of the projects will be the SBT plugins that should be included into other projects (but not to all of them).

Now with these requirements in mind, what should be the structure of the build? Because of requirement 3, I can't just go with a single build.sbt in a root of a build, because I don't want to put all the projects settings there, since it will be a lot of text, and every change in a single project will be reflected on the top level.

I've also heard the usage of *.sbt files both for root project and sub-projects is error-prone and not generally recommended (Producing no artifact for root project with package under multi-project build in SBT, or How can I use an sbt plugin as a dependency in a multi-project build?, SBT: plugins.sbt in subproject is ignored? etc.). I've tried only simple multi-projects builds with *.sbt files on different levels, and it just worked. Which pitfalls do I need to keep in mind if I'll go for a multi *.sbt files approach, given the requirements above?

Answer 1

Ok, since no one have posted anything so far, I've figured I post what I've learned from doing mono repository with SBT so far.

Some facts first: our SBT build consists of 40+ projects, with 10+ common projects (in the sense that other projects depend on them), and 5 groups of projects related to a single product (5-7 projects in each group). In each group, there's typically one group-common project.

Build organization

We have the following build structure:

1. One main build.sbt for the whole build.
2. One build.sbt per project.
3. Several local SBT plugins in project directory.

Let's talk about each of these items.

1. Main build.sbt

In this file, the general build structure is defined. Namely, we keep cross-project dependencies in there. We don't use the standard commonSettings approach, as in:

val commonSettings = Seq(scalaVersion := "2.12.3", ...)
...
val proj1 = (project in file("p1")).settings(commonSettings)
val proj2 = (project in file("p2")).settings(commonSettings)
...

This would be too wordy and easy to get wrong for a new project. Instead we use a local SBT project that automatically applies to every project in the build (more on this later).

2. Per-project build.sbt

In those files, we generally define all the project settings (non-auto plugins, library dependencies, etc.). We don't define cross-project dependencies in these files, because this doesn't really work as expected. SBT loads all the *.sbt files in certain order, and project definition in every build overrides the previously found ones. In other words, if you avoid (re-)defining projects in per-project *.sbt files, things will work well. All the other settings can be kept there, to avoid too much clutter in main build.sbt.

3. Local SBT plugins

We use a trick to define SBT auto-plugin in <root_dir>/project/ directory, and make them load automatically for all the projects in the build. We use those plugins to automatically define the settings and tasks for all the projects (for things like Scalastyle, scalafmt, Sonar, deployment, etc.). We also keep common settings there (scalaVersion, etc.). Another thing we keep in <root_dir>/project/ is common dependencies versions (not in a plugin, just pure *.scala file).

Summary

Using SBT for a mono repository seems to work, and has certain advantages and disadvantages.

Advantages: It's super easy to re-use the code between products. Also common SBT stuff like Scalastyle, scalafmt, etc. is defined once, and all new projects get it for free. Upgrading a dependency version is done in one place for all the projects, so when someone upgrades the version, he or she does this for all the projects at once, so that different teams benefit from that. This requires certain discipline between teams, but it worked for us so far.

Another advantage is use of common tooling. We have a Gerrit+Jenkins continuous integration, and we have a single job for e.g. pre-submit verification. New projects get a lot of this machinery pretty much for free, again.

Disadvantages: For one, the build load time. On top 13" MacBook Pro it can easily last 30+ seconds (this is time from starting SBT to getting to SBT's command prompt). This is not that bad if you can keep the SBT constantly running though. It's much worse for Intellij refreshing the build information, where it can take around 15 minutes. I don't know why it takes so much longer than in SBT, but here's that. Can be mitigated by avoiding refreshing the Intellij unless absolutely necessary, but it's a real pain.

Yet another problem is that you can't load an individual project or group of projects into Intellij IDEA. You are forced to load the build of whole mono repository. If that would be possible, then, I guess, Intellij's situation could have been better.

Another disadvantage is the fact that one can't use different versions of same SBT plugin for different projects. When one project can't be upgraded to a new plugin version for some reason, the whole repository has to wait. Sometimes this is useful, that is, it expedites maintenance work, and forces us to keep projects in maintenance mode up to date. But sometimes for legacy projects it can be challenging.

Conclusion

All in all, we have worked for a around a year in this mode, and we intend to keep doing so in the foreseeable future. What concerns us is the long Intellij IDEA refresh time, and it only gets worse as we add more projects into this build. We might evaluate alternative build systems later that would avoid us loading projects in isolation to help with Intellij performance, but SBT seems to be up to a task.