java.lang.NoSuchMethodError when I try to parse Json on spark

Question

I've got an issues when i'm trying to use com.typesafe.play play-json 2.4.0 on spark. The follwing code make an exception on the spark server but it works perfectly on my pc.

val json = Json.parse(json_string)

the exeption :

java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonToken.id()I
    at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:122)
    at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:108)
    at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:103)
    at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:2860)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1569)
    at play.api.libs.json.jackson.JacksonJson$.parseJsValue(JacksonJson.scala:226)
    at play.api.libs.json.Json$.parse(Json.scala:21)
    at org.soprism.kafka.connector.TwitterToCassandraPostsParser$.ParseJson(TwitterToCassandraPostsParser.scala:74)
    at org.soprism.kafka.connector.TwitterToCassandraPostsParser$$anonfun$1$$anonfun$apply$1.apply(TwitterToCassandraPostsParser.scala:65)
    at org.soprism.kafka.connector.TwitterToCassandraPostsParser$$anonfun$1$$anonfun$apply$1.apply(TwitterToCassandraPostsParser.scala:65)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:798)
    at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:798)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1503)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1503)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
    at org.apache.spark.scheduler.Task.run(Task.scala:64)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

I use a spark-submit command to execute it

It seems to be an incompatibily between two versions of jackson's library. How can I fix it ?

Thank you

Answer 1

Spark nodes will NOT check your dependencies. You need to build an uber-jar that includes all your dependencies and pass it to Spark in order to be distributed to the different node.