Menu
My sbt project takes more than 15 minutes when I do
sbt clean compile
I am on a beefy machine on AWS. I am fairly certain its not a resource issue on cpu or internet bandwidth. Also, I have run this command a few times and hence the ivy cache is populated.
Here is all my build related files
/build.sbt
name := "ProjectX"
version := "1.0"
scalaVersion := "2.10.5"
libraryDependencies += ("org.apache.spark" %% "spark-streaming" % "1.4.1")
.exclude("org.slf4j", "slf4j-log4j12")
.exclude("log4j", "log4j")
.exclude("commons-logging", "commons-logging")
.%("provided")
libraryDependencies += ("org.apache.spark" %% "spark-streaming-kinesis-asl" % "1.4.1")
.exclude("org.slf4j", "slf4j-log4j12")
.exclude("log4j", "log4j")
.exclude("commons-logging", "commons-logging")
libraryDependencies += "org.mongodb" %% "casbah" % "2.8.1"
//test
libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.4" % "test"
//logging
libraryDependencies ++= Seq(
//facade
"org.slf4j" % "slf4j-api" % "1.7.12",
"org.clapper" %% "grizzled-slf4j" % "1.0.2",
//jcl (used by aws sdks)
"org.slf4j" % "jcl-over-slf4j" % "1.7.12",
//log4j1 (spark)
"org.slf4j" % "log4j-over-slf4j" % "1.7.12",
//log4j2
"org.apache.logging.log4j" % "log4j-api" % "2.3",
"org.apache.logging.log4j" % "log4j-core" % "2.3",
"org.apache.logging.log4j" % "log4j-slf4j-impl" % "2.3"
//alternative to log4j2
//"org.slf4j" % "slf4j-simple" % "1.7.5"
)
/project/build.properties
sbt.version = 0.13.8
/project/plugins.sbt
logLevel := Level.Warn
addSbtPlugin("org.scalastyle" %% "scalastyle-sbt-plugin" % "0.7.0")
resolvers += "sonatype-releases" at "https://oss.sonatype.org/content/repositories/releases/"
/project/assembly.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")
1000
Library dependencies can be added in two ways: unmanaged dependencies are jars dropped into the lib directory. managed dependencies are configured in the build definition and downloaded automatically from repositories.
Like in almost every build system, SBT allows you to define library dependencies which are resolved automatically, so you don't have to download and package required libraries by yourself.
If you have JAR files (unmanaged dependencies) that you want to use in your project, simply copy them to the lib folder in the root directory of your SBT project, and SBT will find them automatically.
You can use both managed and unmanaged dependencies in your SBT projects. If you have JAR files (unmanaged dependencies) that you want to use in your project, simply copy them to the lib folder in the root directory of your SBT project, and SBT will find them automatically.
On the log do you see entries like:
[info] [SUCCESSFUL ] org.apache.spark#spark-streaming-kinesis-asl_2.10;1.4.1!spark-streaming-kinesis-asl_2.10.jar (239ms)
That's a sign that you're downloading these artifacts. In other words, the AMI you're launching doesn't have the Ivy cache populated.
Using sbt 0.13.12 on my laptop with SSD, I get about 5s for clean and then update.
so-31956971> update
[info] Updating {file:/xxx/so-31956971/}app...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[success] Total time: 5 s, completed Aug 25, 2016 4:00:00 AM
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
