Why does sbt compile fail with StackOverflowError?

Question

I am working on a Scala project that has been in existence for a few years but is new to me. My task is to upgrade it from Scala 2.9.3 to 2.11.7, along with its dependencies. I have gotten past the errors and warnings, but I cannot get the project to compile successfully in SBT. I always get a StackOverflowError in pretty much the same place. The stacktrace looks like this, but details vary with the Xss setting (currently 4M, but have tried as high as 24M):

java.lang.StackOverflowError
at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:698)
at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5395)
at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5422)
at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5369)
at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5373)
at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5471)
at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5479)
at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:644)
at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:698)
at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5395)
at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5422)

SBT_OPTS looks like this:

-Xmx2G -Xss4M -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled

I can 'make' the project successfully in Intellij, and others can pull my changes from GitHub and compile the project in sbt, so the issue seems to be local to my machine (a recent quad-core Macbook Pro with 16GB RAM). Other Scala/sbt projects compile successfully for me on this machine.

Here are other relevant details:

Scala version: 2.11.7
Java version: java version "1.8.0_66" (build 1.8.0_66-b17)
sbt version: 0.13.7 (have also tried 0.13.9)

I have completely rebuilt the ivy2 cache and cleared the lib_managed directory. The version of the scala-compiler.jar is the same as is used on at least one machine that can 'sbt compile' the code successfully. I did a clean reinstall of sbt (via brew remove sbt, manual removal of ~/.sbt directory, then brew install sbt).

I have not tried to isolate the line of source code being compiled when the error occurs. I have assumed it would be more productive to look for a configuration issue or dependency conflict somewhere.

Any suggestions for further troubleshooting will be appreciated.

[Added...] It may be helpful to add that, as an experiment, I downloaded the Scala language source code from https://github.com/scala/scala and got the following very similar error trying to sbt compile it:

java.lang.StackOverflowError
at scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.outerValue(ExplicitOuter.scala:229)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:441)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:352)
at scala.reflect.internal.Trees$class.itransform(Trees.scala:1345)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)
at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.transform(TypingTransformers.scala:44)
at scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.scala$reflect$internal$Trees$UnderConstructionTransformer$$super$transform(ExplicitOuter.scala:219)
at scala.reflect.internal.Trees$UnderConstructionTransformer$class.transform(Trees.scala:1693)
at scala.tools.nsc.transform.ExplicitOuter$OuterPathTransformer.transform(ExplicitOuter.scala:291)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:459)
at scala.tools.nsc.transform.ExplicitOuter$ExplicitOuterTransformer.transform(ExplicitOuter.scala:352)
at scala.reflect.internal.Trees$class.itransform(Trees.scala:1347)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:16)
at scala.reflect.api.Trees$Transformer.transform(Trees.scala:2555)

Here is something interesting. From this post I found out about launching sbt with a -d flag for debugging info. Got the following output:

Kevins-MacBook-Pro:scala kdoherty$ sbt -d
[process_args] java_version = '1.8.0_66'
# Executing command line:
java
-Xmx2G
-Xss4M
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-Xmx384m
-Xss512k
-XX:+UseCompressedOops
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
-jar
/usr/local/Cellar/sbt/0.13.9/libexec/sbt-launch.jar

So somewhere my SBT_OPTS settings are being overridden (by defaults, I guess). Now I need to find where those defaults are coming from.

Answer 1

I figured it out. Once I knew that the -d flag would tell me what settings SBT was actually using, I saw that the values in my SBT_OPTS environment variable were being clobbered by other, lower settings. Where were those coming from? From my JAVA_OPTS env variable! I should have noticed them sooner, but now I know I can keep those Java options as they are and override them by adding the SBT-specific settings to my /usr/local/etc/sbtopts file, using the somewhat awkward format of

-J-Xmx2G -J-Xss2M  

Using the values shown I was able to run sbt compile successfully on my project.

I hope someone finds this useful.

Answer 2

Add to the bottom of /usr/local/etc/sbtopts

-J-Xmx4G  -J-Xss4M 

All set.

Answer 3

I just added -Xss in my Intellij sbt properties and the issue is resolved.

Intellij SBT properties

Answer 4

Relevant parts from the output of sbt -h:

  # jvm options and output control
  JAVA_OPTS          environment variable, if unset uses ""
  .jvmopts           if this file exists in the current directory, its contents
                     are appended to JAVA_OPTS
  SBT_OPTS           environment variable, if unset uses ""
  .sbtopts           if this file exists in the current directory, its contents
                     are prepended to the runner args

Thus, in my case I solved the problem by creating a file .sbtopts with content

-J-Xmx4G 
-J-Xss4M

in the project directory.

Note: Running sbt -d shows what settings have been used, for instance:

$ sbt -d
[addSbt] arg = '-debug'
[process_args] java_version = '8'
# Executing command line:
java
-Xms1024m
-XX:ReservedCodeCacheSize=128m
-XX:MaxMetaspaceSize=256m
-Xmx2G
-Xss2M
-jar
/path/to/sbt-launch.jar
-debug