java.lang.ClassNotFoundException: org.apache.spark.sql.Dataset

Question

When running a Scala file that uses the Spark Dataset type I get the following stack trace:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/Dataset
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:125)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.Dataset
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 6 more

I find this strange because I have the following import:

import org.apache.spark.sql._

Also, in my build.sbt I have the following added to libraryDependencies:

  "org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
  "org.apache.spark" %% "spark-sql" % "1.6.2" % "provided",

Answer 1

If you are executing this standalone you can try removing provided from your dependencies. Provided means that you expect the dependencies to already be on the classpath when you run this application. So the Spark dependencies won't be included in your jar if you use provided.

Answer 2

In IntelliJ 2020.3.2 community edition, go to menu run then edit configurations. Finally, in Modify options select 'Include dependencies with "Provided" scope'.