When to use "sbt assembly" and "sbt compile && sbt package"?

Question

I would like to know when should I use sbt assembly and when sbt compile && sbt package?

I'm coding a program in my local computer using Intellij IDEA and I'm compiling it with sbt compile && sbt package, then I'm going to upload it to a cluster and run it using spark-submit. Is this the best way? When should I use sbt assembly?

Answer 1

I would like to know when should I use sbt assembly and when sbt compile && sbt package?

Quoting sbt-assembly's website (which gives sbt assembly):

The goal is simple: Create a fat JAR of your project with all of its dependencies.

Any dependency in build.sbt (that is not marked as Provided) is going to be part of the final jar file that is called an uber-jar.

There are at least two reasons why you'd like to use the plugin:

1. Have all dependencies bundled together in one jar file that is much easier to deploy to production (akin to a Docker image)
2. Run your application with one Spark version, say 2.0.0, with a cluster that has spark-submit from Spark 1.6.0

You use sbt compile && sbt package to compile and package the classes of your application (which by the way does not have to use Spark whatsoever). That jar file does not include the dependencies of your application so any dependencies have to be provided in some other way (and makes deployment a bit tricker).

NB: sbt compile && sbt package is just a longer variant of sbt compile package which given that compile task is a dependency of package is just sbt package.