Spark streaming elasticsearch dependencies

Question

Im trying Spark & Elasticsearch integration in scala as described in Elasticsearch Guide

I have problems with dependencies when compiling :

[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: cascading#ing-local;2.5.6: not found
[error] unresolved dependency: clj-time#clj-time;0.4.1: not found
[error] unresolved dependency: compojure#compojure;1.1.3: not found
[error] unresolved dependency: hiccup#hiccup;0.3.6: not found
[error] unresolved dependency: ring#ring-devel;0.3.11: not found
[error] unresolved dependency: ring#ring-jetty-adapter;0.3.11: not found
[error] unresolved dependency: com.twitter#carbonite;1.4.0: not found
[error] unresolved dependency: cascading#cascading-hadoop;2.5.6: not found
[error] Total time: 86 s, completed 19 nov. 2014 08:42:58

My build.sbt file look like this

name := "twitter-sparkstreaming-elasticsearch"

version := "0.0.1"

scalaVersion := "2.10.4"

// additional libraries
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "1.1.0",
  "org.apache.spark" %% "spark-streaming" % "1.1.0",
  "org.apache.spark" %% "spark-streaming-twitter" % "1.1.0",
  "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0"
)

Help? thanks.

Answer 1

Cascading and its dependencies are not available in Maven central but in their own repo (which es-hadoop cannot specify through its pom).

I resolved the problem by using elasticsearch-spark_2.10

http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/install.html