Decreasing the compilation time when using shapeless HList

Question

Scala 2.12.6, Shapeless 2.3.3.

I have a lot of large models (case classes) which are quite deep. I use shapeless to help use/manipulate these models and also use libraries like circe which make heavy use of shapeless.

This has caused my compilation time to increase massively during the phase typer part of scalac.

Based on a bit of Googling, it appears that shapeless is the culprit, but I can't seem to find any concrete tips on how I can improve this.

It was suggested that because I am resolving the HList implicits multiple times (because of the multiple libraries) for the same models, that I should "cache them" - however I'm not sure how to figure out exactly what to cache.

Given something like:

case class MyModel(value: String) extends AnyVal
case class MyOtherModel(value: Int) extends AnyVal
case class MyRootModel(myModel: MyModel, myOtherModel: MyOtherModel)

What should I be caching for MyModel/MyOtherModel and MyRootModel?

Answer 1

You can cache Shapeless's LabelledGeneric instances like this:

import shapeless.{LabelledGeneric, the}

case class MyModel(value: String) extends AnyVal
case class MyOtherModel(value: Int) extends AnyVal
case class MyRootModel(myModel: MyModel, myOtherModel: MyOtherModel)

object MyModel {
  implicit val generic = the[LabelledGeneric[MyModel]]
}

object MyOtherModel {
  implicit val generic = the[LabelledGeneric[MyOtherModel]]
}

object MyRootModel {
  implicit val generic = the[LabelledGeneric[MyRootModel]]
}

You should of course see if this improves compilation times in your own project, but as a quick benchmark we can set up a test that resolves the LabelledGeneric repeatedly (a thousand times in this case):

object Test {
  def foo0 = {
    LabelledGeneric[MyRootModel]
    LabelledGeneric[MyRootModel]
    // repeat 98 more times...
  }
  def foo1 = {
    LabelledGeneric[MyRootModel]
    LabelledGeneric[MyRootModel]
    // repeat 98 more times...
  }
  // and so on through foo9 
}

(Note that we have to split up the invocations because if we just dumped a thousand of them in a row in a single method the macro-generated code would exceed the JVM's method size limits when we comment out the instance caching to compare.)

On my machine a Test.scala file containing Test, the case class definitions, and the cached instances compiles in about 3 seconds. If we comment out the generic definitions, it takes over 12 seconds. This is of course pretty unscientific, but it's encouraging.

Note that in general it's not a good idea to have implicit definitions without a type annotation, and you can avoid doing that for the cached instances by writing something like this:

import shapeless.{LabelledGeneric, TypeOf, cachedImplicit}

case class MyModel(value: String) extends AnyVal
case class MyOtherModel(value: Int) extends AnyVal
case class MyRootModel(myModel: MyModel, myOtherModel: MyOtherModel)

object MyModel {
  implicit val generic: TypeOf.`LabelledGeneric[MyModel]`.type = cachedImplicit
}

object MyOtherModel {
  implicit val generic: TypeOf.`LabelledGeneric[MyOtherModel]`.type = cachedImplicit
}

object MyRootModel {
  implicit val generic: TypeOf.`LabelledGeneric[MyRootModel]`.type = cachedImplicit
}

TypeOf is some weird magic, though, and to be honest when I've needed something like this I've just used the the approach.

As a footnote, since you mention circe specifically, you might want to give circe-derivation a try. It works as a drop-in replacement for much of the functionality of circe-generic, but isn't built on Shapeless and compiles much more quickly.