Running spark application in local mode

Question

I'm trying to start my Spark application in local mode using spark-submit. I am using Spark 2.0.2, Hadoop 2.6 & Scala 2.11.8 on Windows. The application runs fine from within my IDE (IntelliJ), and I can also start it on a cluster with actual, physical executors.

The command I'm running is

spark-submit --class [MyClassName] --master local[*] target/[MyApp]-jar-with-dependencies.jar [Params]

Spark starts up as usual, but then terminates with

java.io.Exception: Failed to connect to /192.168.88.1:56370

What am I missing here?

Answer 1

Check which port you are using: if on cluster: log in to master node and include:

--master spark://XXXX:7077

You can find it always in spark ui under port 8080

Also check your spark builder config if you have set master already as it takes priority when launching eg:

val spark = SparkSession
  .builder
  .appName("myapp")
  .master("local[*]")