Can't start Apache Spark on Windows using Cygwin

Question

I am sorry if this is too trivial but I am very new to Spark. I built Apache Spark on Windows from code using Maven. I used Cygwin to build and it worked fine. Now I am unable to start it using the command (./start-master.sh) from the sbin directory.

I get the below error : failed to launch org.apache.spark.deploy.master.Master:

Error: Could not find or load main class org.apache.spark.launcher.Main
full log in /cygdrive/c/Spark/spark-1.5.1/sbin/../logs/spark-auser-org.apache.spark.deploy.master.Master-1.host

I understand it is looking for the spark-launcher_2.10-1.5.1.jar in the classpath but I am not able to identify in the scripts where is it defining the classpath or the lib directory.

Answer 1

You could follow the Building a Runnable Distribution steps as Gimby mentioned, like this for example:

./make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.4 -Phive -Phive-thriftserver -Pyarn

---

As Patrick McGloin mentioned, you could also start a Spark Cluster on Windows itself, like this for example:

spark-class.cmd org.apache.spark.deploy.master.Master
spark-class.cmd org.apache.spark.deploy.worker.Worker spark://localhost:7077

You can read more in Spark Standalone Mode and How to set up Spark cluster on Windows machines?

---

Finally as OP mentioned, (s)he was able to built a runnable distribution that created a new Custom-spark tar file in the parent directory. (S)he took this file, untarred it in a different directory, went to the bin subdirectory and then was able to bring up the scala-shell.