How to submit a Scala job to Spark?

Question

I have a Pythons script that I was able to submit to Spark in the following way:

/opt/spark/bin/spark-submit --master yarn-client test.py

Now, I try to submit a Scala program in the same way:

/opt/spark/bin/spark-submit --master yarn-client test.scala

As a result I get the following error message:

Error: Cannot load main class from JAR file:/home/myname/spark/test.scala
Run with --help for usage help or --verbose for debug output

The Scala program itself is just a Hello World program:

object HelloWorld {
    def main(args: Array[String]): Unit = {
        println("Hello, world!")
    }
}

What am I doing wrong?

Answer 1

For starters you'll have to create a jar file. You cannot simply submit Scala source. If in doubt see Getting Started with sbt.

After that just add a class parameter pointing to the HelloWorld. Assuming no packages:

/opt/spark/bin/spark-submit --master yarn-client --class "HelloWorld" path_to.jar

Answer 2

It depends on cluster mode you are using.

Have a look at generic command

./bin/spark-submit \
  --class <main-class>
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

For yarn-client,

/opt/spark/bin/spark-submit \
  --class "HelloWorld" your_jar_with_scala_file \
  --master yarn-client

Have a look at Spark documentation for better understanding.