Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to build an Uber JAR (Fat JAR) using SBT within IntelliJ IDEA?

I'm using SBT (within IntelliJ IDEA) to build a simple Scala project.

I would like to know what is the simplest way to build an Uber JAR file (aka Fat JAR, Super JAR).

I'm currently using SBT but when I'm submiting my JAR file to Apache Spark I get the following error:

Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes

Or this error during compilation time:

java.lang.RuntimeException: deduplicate: different file contents found in the following:
PATH\DEPENDENCY.jar:META-INF/DEPENDENCIES
PATH\DEPENDENCY.jar:META-INF/MANIFEST.MF

It looks like it is because some of my dependencies include signature files (META-INF) which needs to be removed in the final Uber JAR file.

I tried to use the sbt-assembly plugin like that:

/project/assembly.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0") 

/project/plugins.sbt

logLevel := Level.Warn 

/build.sbt

lazy val commonSettings = Seq(   name := "Spark-Test"   version := "1.0"   scalaVersion := "2.11.4" )  lazy val app = (project in file("app")).   settings(commonSettings: _*).   settings(     libraryDependencies ++= Seq(       "org.apache.spark" %% "spark-core" % "1.2.0",       "org.apache.spark" %% "spark-streaming" % "1.2.0",       "org.apache.spark" % "spark-streaming-twitter_2.10" % "1.2.0"     )   ) 

When I click "Build Artifact..." in IntelliJ IDEA I get a JAR file. But I end up with the same error...

I'm new to SBT and not very experimented with IntelliJ IDE.

Thanks.

like image 792
Yves M. Avatar asked Feb 11 '15 16:02

Yves M.


People also ask

How do I run a JAR file in sbt?

Solution. Create a directory layout to match what SBT expects, then run sbt compile to compile your project, sbt run to run your project, and sbt package to package your project as a JAR file. Unlike Java, in Scala, the file's package name doesn't have to match the directory name.

What is uber jar in Java?

An uber-JAR—also known as a fat JAR or JAR with dependencies—is a JAR file that contains not only a Java program, but embeds its dependencies as well. This means that the JAR functions as an “all-in-one” distribution of the software, without needing any other Java code.


1 Answers

Finally I totally skip using IntelliJ IDEA to avoid generating noise in my global understanding :)

I started reading the official SBT tutorial.

I created my project with the following file structure :

my-project/project/assembly.sbt my-project/src/main/scala/myPackage/MyMainObject.scala my-project/build.sbt 

Added the sbt-assembly plugin in my assembly.sbt file. Allowing me to build a fat JAR :

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0") 

My minimal build.sbt looks like :

lazy val root = (project in file(".")).   settings(     name := "my-project",     version := "1.0",     scalaVersion := "2.11.4",     mainClass in Compile := Some("myPackage.MyMainObject")           )  val sparkVersion = "1.2.0"  libraryDependencies ++= Seq(   "org.apache.spark" %% "spark-core" % sparkVersion % "provided",   "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",   "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion )  // META-INF discarding mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>    {     case PathList("META-INF", xs @ _*) => MergeStrategy.discard     case x => MergeStrategy.first    } } 

Note: The % "provided" means not to include the dependency in the final fat JAR (those libraries are already included in my workers)

Note: META-INF discarding inspired by this answser.

Note: Meaning of % and %%

Now I can build my fat JAR using SBT (how to install it) by running the following command in my /my-project root folder:

sbt assembly 

My fat JAR is now located in the new generated /target folder :

/my-project/target/scala-2.11/my-project-assembly-1.0.jar 

Hope that helps someone else.


For those who wants to embeed SBT within IntelliJ IDE: How to run sbt-assembly tasks from within IntelliJ IDEA?

like image 145
Yves M. Avatar answered Sep 26 '22 02:09

Yves M.