NoClassDefFound : Scala/xml/metadata

Question

I'm running a simple spark program in Java (IDE : Eclipse Luna, Maven).

My Sample program is

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;

public class testSpark {
    public static void main(String[] args) {
        SparkConf conf = new SparkConf().setAppName("Testing").setMaster("local");
        JavaSparkContext sc = new JavaSparkContext(conf);
        
        System.out.println(sc.appName());
    }
}

But I'm getting error while running my sample program

Exception in thread "main" java.lang.NoClassDefFoundError:
scala/xml/MetaData  at
  org.apache.spark.ui.jobs.JobsTab.<init>(JobsTab.scala:30)     at
  org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:50)  at
  org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:61)  at
  iScope.testSpark.main(testSpark.java:9) 

Caused by:
  java.lang.ClassNotFoundException: scala.xml.MetaData  at
  java.net.URLClassLoader.findClass(Unknown Source)     at
  java.lang.ClassLoader.loadClass(Unknown Source)

My pom.xml file is

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>1.2.1</version>
</dependency>
<dependency>
    <groupId>com.google.guava</groupId>
    <artifactId>guava</artifactId>
    <version>18.0</version>
    </dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-mllib_2.11</artifactId>
    <version>1.2.1</version>
</dependency>

Answer 1

SparkUI seems to use a scala package, try adding this dsependency to your pom file to put the scala.xml package on your classpath.

<dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-xml</artifactId>
    <version>2.11.0-M4</version>
</dependency>

Answer 2

Adding the following dependency fixed the issue for me.

<dependency>
    <groupId>org.scala-lang.modules</groupId>
    <artifactId>scala-xml_2.11</artifactId>
    <version>1.0.6</version>
</dependency>

Answer 3

As your exception's stack trace clearly states, the problem happens when SparkUI tries to build up the Jobs Tab. For that to happen, the JobsTab class (org.apache.spark.ui.jobs.JobsTab) tries to create a page (org.apache.spark.ui.jobs.JobPage) and attach it to itself. If you look into the JobPage source code you will note that it makes heavy use of scala.xml (The Standard Scala XML Library), which you are probably missing.

As already pointed out by other contributors, adding the scala.xml library to the list of your dependencies should fix the problem. At the time of writing, the latest version is 1.2.0 for Scala 2.13 (you can check the Maven repository for updates), hence:

Maven:
<dependency>
    <groupId>org.scala-lang.modules</groupId>
    <artifactId>scala-xml_2.13</artifactId>
    <version>1.2.0</version>
</dependency>

Gradle:
compile group: 'org.scala-lang.modules', name: 'scala-xml_2.13', version: '1.2.0'

SBT:
libraryDependencies += "org.scala-lang.modules" %% "scala-xml" % "1.2.0"