sbt won't assemble Spark

Question

I am having issues assembling Spark using the sbt on my machine. Attempting the assembly without allocating extra memory either runs out or times out on the garbage collector; the exact issue has been different at different times I have tried. However, any attempt to modify the allocated memory, either through Xmx or Xms, and whether giving more or restricting, fails as the sbt doesn't recognize -Xmx or -Xms as a command.

Here is a sample of the kind of command I have been trying (from the source directory of Spark):

sbt -Xmx2g assembly

Here is the error I have been receiving:

java.util.concurrent.ExecutionException:java.lang.OutOfMemoryError: GC overhead limit exceeded
Use 'last' for the full log.
Not a valid command: Xmx2g
Not a valid project: Xmx2g
Expected ':' (if selecting a configuration)
Not a valid key: Xmx2g
Xmx2g
     ^

I am running 64-bit Java, version 1.8.0_20.

Answer 1

Try creating a new environment variable SBT_OPTS, with the value "-XX:MaxPermSize=1024m". That should give sbt the memory it needs without producing your error.