Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

UNRESOLVED DEPENDENCIES error while trying to create jar

Tags:

I'm trying to build a Scala jar file to run it in spark.
I'm following this tutorial.
when trying to build jar file using sbt as here, i'm facing with following error

[info] Resolving org.apache.spark#spark-core_2.10.4;1.0.2 ... [warn]  module not found: org.apache.spark#spark-core_2.10.4;1.0.2 [warn] ==== local: tried [warn]   /home/hduser/.ivy2/local/org.apache.spark/spark-core_2.10.4/1.0.2/ivys/ivy.xml [warn] ==== Akka Repository: tried [warn]   http://repo.akka.io/releases/org/apache/spark/spark-core_2.10.4/1.0.2/spark-core_2.10.4-1.0.2.pom [warn] ==== public: tried [warn]   http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10.4/1.0.2/spark-core_2.10.4-1.0.2.pom [warn]  :::::::::::::::::::::::::::::::::::::::::::::: [warn]  ::          UNRESOLVED DEPENDENCIES         :: [warn]  :::::::::::::::::::::::::::::::::::::::::::::: [warn]  :: org.apache.spark#spark-core_2.10.4;1.0.2: not found [warn]  :::::::::::::::::::::::::::::::::::::::::::::: [error] {file:/home/prithvi/scala/asd/}default-d57abf/*:update: sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.10.4;1.0.2: not found [error] Total time: 2 s, completed 13 Aug, 2014 5:24:24 PM 

what's the issue and how to solve it.


Dependency issue has been resolved. Thank you "om-nom-nom"
but new error arised

[warn]  :::::::::::::::::::::::::::::::::::::::::::::: [warn]  ::              FAILED DOWNLOADS            :: [warn]  :: ^ see resolution messages for details  ^ :: [warn]  :::::::::::::::::::::::::::::::::::::::::::::: [warn]  :: org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645!javax.transaction.orbit [warn]  :: org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016!javax.servlet.orbit [warn]  :: org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020!javax.mail.glassfish.orbit [warn]  :: org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233!javax.activation.orbit [warn]  :::::::::::::::::::::::::::::::::::::::::::::: [error] {file:/home/prithvi/scala/asd/}default-c011e4/*:update: sbt.ResolveException: download failed: org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645!javax.transaction.orbit [error] download failed: org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016!javax.servlet.orbit [error] download failed: org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020!javax.mail.glassfish.orbit [error] download failed: org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233!javax.activation.orbit [error] Total time: 855 s, completed 14 Aug, 2014 12:28:33 PM 
like image 765
Y.Prithvi Avatar asked Aug 13 '14 12:08

Y.Prithvi


2 Answers

You have your dependency defined as

"org.apache.spark" %% "spark-core" % "1.0.2" 

That %% instructs sbt to substitute current scala version to artifact name. Apparently, spark was build for the whole family of 2.10 scala, without specific jars for 2.10.1, 2.10.2 ...

So all you have to do is to redefine it as:

"org.apache.spark" % "spark-core_2.10" % "1.0.2" 
like image 171
om-nom-nom Avatar answered Sep 29 '22 08:09

om-nom-nom


I had the same issue. Looks like that some bugs are in different versions/compilations/etc.

For me the following build.sbt worked fine

name := "My Project"   version := "1.0"   scalaVersion := "2.11.8"   libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.2"   

Hope it helps

like image 26
Danylo Zherebetskyy Avatar answered Sep 29 '22 06:09

Danylo Zherebetskyy