Menu
Same problem as Failed to start master for spark in windows 10 which is also not solved.
My spark is working well by testing pyspark.cmd and spark-shell.cmd
After runing .\sbin\start-master.sh I got:
ps: unknown option -- o
Try 'ps --help' for more information.
starting org.apache.spark.deploy.master.Master, logging to C:\spark-1.6.1-bin-hadoop2.6/logs/spark--org.apache.spark.deploy.master.Master-1-%MY_USER_NAME%-PC.out
ps: unknown option -- o
Try 'ps --help' for more information.
failed to launch org.apache.spark.deploy.master.Master:
========================================
Picked up _JAVA_OPTIONS: -Xmx512M -Xms512M
full log in C:\spark-1.6.1-bin-hadoop2.6/logs/spark--org.apache.spark.deploy.master.Master-1-%MY_USER_NAME%-PC.out
I tried to visit web UI, while the localhost:4040 is working the localhost:8080 cannot be reached.
And I found there is the .log file created at the folder of %SPARK_HOME%/logs . They contains same content:
Spark Command:
C:\Program Files\Java\jdk1.7.0_79\bin\java -cp C:\spark-1.6.1-bin-hadoop2.6/conf\;C:\spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar;C:\spark-1.6.1-bin-hadoop2.6\lib\datanucleus-api-jdo-3.2.6.jar;C:\spark-1.6.1-bin-hadoop2.6\lib\datanucleus-core-3.2.10.jar;C:\spark-1.6.1-bin-hadoop2.6\lib\datanucleus-rdbms-3.2.9.jar -Xms1g -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.master.Master --ip hahaha-PC --port 7077 --webui-port 8080
========================================
Picked up _JAVA_OPTIONS: -Xmx512M -Xms512M
Working environment: Spark: 1.6.1 Windows 10
Looking forward to your reply and thanks for your time so much!
304
Go to spark installation folder, open Command Prompt as administrator and run the following command to start master node. The host flag ( --host ) is optional. It is useful to specify an address specific to a network interface when multiple network interfaces are present on a machine.
Click Analytics > Spark Analytics > Open the Spark Application Monitoring Page. Click Monitor > Workloads, and then click the Spark tab. This page displays the user names of the clusters that you are authorized to monitor and the number of applications that are currently running in each cluster.
Connecting an Application to the Cluster To run an application on the Spark cluster, simply pass the spark://IP:PORT URL of the master as to the SparkContext constructor. You can also pass an option --total-executor-cores <numCores> to control the number of cores that spark-shell uses on the cluster.
spark-shell is a CLI utility that comes with Apache Spark distribution, open command prompt, go to cd %SPARK_HOME%/bin and type spark-shell command to run Apache Spark shell. You should see something like below (ignore the error you see at the end).
The launch scripts located at %SPARK_HOME%\sbin do not support Windows. You need to manually run the master and worker as outlined below.
Go to %SPARK_HOME%\bin folder in a command prompt
Run spark-class org.apache.spark.deploy.master.Master to run the master. This will give you a URL of the form spark://ip:port
Run spark-class org.apache.spark.deploy.worker.Worker spark://ip:port to run the worker. Make sure you use the URL you obtained in step 2.
Run spark-shell --master spark://ip:port to connect an application to the newly created cluster.
57
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
