How to override dependency on certain task in sbt

Question

I want to override dependency on project in certain Task. I have a sbt multi-project which using spark.

lazy val core = // Some Project

val sparkLibs = Seq(
  "org.apache.spark" %% "spark-core" % "1.6.1"
)

val sparkLibsProvided = Seq(
  "org.apache.spark" %% "spark-core" % "1.6.1" % "provided"
)

lazy val main = Project(
  id = "main",
  base = file("main-project"),
  settings = sharedSettings
).settings(
  name := "main",
  libraryDependencies ++= sparkLibs,
  dependencyOverrides ++= Set(
    "com.fasterxml.jackson.core" % "jackson-databind" % "2.4.4"
  )
).dependsOn(core)

When I try to make fat jar to submit on my yarn cluster, I use https://github.com/sbt/sbt-assembly task. But in this case, I want to use sparkLibsProvided instead of sparkLibs something like:

lazy val sparkProvided = (project in assembly).settings(
  dependencyOverrides ++= sparkLibsProvided.toSet
)

How can I properly override this dependency?

Answer 1

You can create a new project which is a dedicated project for creating your spark uber jar with the provided flag:

lazy val sparkUberJar = (project in file("spark-project"))
  .settings(sharedSettings: _*)
  .settings(
    libraryDependencies ++= sparkLibsProvided,
    dependencyOverrides ++= Set(
      "com.fasterxml.jackson.core" % "jackson-databind" % "2.4.4"
    )
  )

And when you assemble in sbt, go to the said project first:

sbt project sparkUberJar
sbt assembly