unable to bring up spark 2.1.0 manually on HDP 2.5.3

Question

I was testing my spark code on spark 2.0.0 and I hit a bug SPARK-17463 and I wanted to use spark 2.1.0 since the bug is fixed in this version.

However, I am unable to bring up spark-shell with yarn client mode for spark 2.1.0

I need to get 2.1.0 working on a HDP 2.5.3 cluster.

It throws an exception:

    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/02/15 14:28:46 WARN Utils: Service 'SparkUI' could not bind on port 4045. Attempting port 4046.
    java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig
      at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
      at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
      at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
      at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
      at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:151)
      at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
      at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
      at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
      at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
      at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
      ... 47 elided
    Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.config.ClientConfig
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 61 more
<console>:14: error: not found: value spark
           import spark.implicits._
                          ^
<console>:14: error: not found: value spark
           import spark.sql
                          ^

Answer 1

I had the same issue with spark-submit in EMR, after adding jersey-bundle-1.19.1.jar to $SPARK_HOME/jars, issue got resolved.

you can download it from here: http://repo1.maven.org/maven2/com/sun/jersey/jersey-bundle/1.19.1/jersey-bundle-1.19.1.jar

Answer 2

The YARN time service is not compatible with libraries provided by Spark. Please disable time service by setting spark.hadoop.yarn.timeline-service.enabled=false.

For more details please visit https://issues.apache.org/jira/browse/SPARK-15343