SBT hangs resolving dependencies

Question

We have an SBT 0.13.0 multi-project build with 17 projects: 1 leaf project, 15 modules that depend on the leaf (but not each other), and 1 aggregator that depends on the 15 modules.

Here's a very rough idea of what the Build.scala file looks like:

val deps: Seq[Setting[_]] = Seq(libraryDependencies ++= Seq(
  "com.foo" % "foo" % "1.0.0",
  "com.bar" % "bar" % "1.0.0"
))

val leaf = Project("leaf").settings(deps:_*)

val module1 = Project("module1").dependsOn(leaf).settings(deps:_*)
val module2 = Project("module2").dependsOn(leaf).settings(deps:_*)
...
val module15 = Project("module15").dependsOn(leaf).settings(deps:_*)

val aggregator = Project("aggregator)".dependsOn(
  module1,
  module2,
  ...
  module15
).settings(deps:_*)

All of these projects list exactly the same set of external dependencies as libraryDependencies. For some reason, when we run the update command in the aggregator, it takes on the order of a minute per project (~15 minutes total!), even though there is no single new dependency to resolve or download.

Worse yet, we recently added one more dependency and now the update command causes SBT to swell up to ~5GB of memory and sometimes hang completely during resolution. How do we debug this?

We tried YourKit to profile it and, it may be a read herring, but so far, the only thing we see is some sbt.MultiLogger class spending a ton of time in a BufferedOutputStream.flush call.

Answer 1

If some of your external dependiencies are actually your own libraries pushed to a local repository and their version is set to "latest" then this hanging is expected. The reason is that ivy tries all repositories for all dependencies when "latest" version is required. Since your libraries are not pushed to public repositories then checking for them on public repositories ends in timeout (it is an ivy issue apparently).

I tried to replicate your setup and created an sbt project with leaf, 15 modules, aggregator and a few external dependencies. It all resolves quite quickly. You can check it out at

https://github.com/darkocerdic/sbt-multiproject-resolving