Menu
Spark version 1.3.0
Error while submitting jobs to spark cluster in cluster mode
./spark-submit --class org.apache.spark.examples.streaming.JavaDirectKafkaWordCount --deploy-mode cluster wordcount-0.1.jar 172.20.5.174:9092,172.20.9.50:9092,172.20.7.135:9092 log
Yields:
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Running Spark using the REST application submission protocol.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/04/14 16:41:10 INFO StandaloneRestClient: Submitting a request to launch an application in spark://172.20.9.151:7077.
Warning: Master endpoint spark://172.20.9.151:7077 was not a REST server. Falling back to legacy submission gateway instead.
15/04/14 16:41:11 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Sending launch command to spark://172.20.9.151:7077
Error connecting to master spark://172.20.9.151:7077 (akka.tcp://[email protected]:7077), exiting.
249
You can submit a Spark batch application by using cluster mode (default) or client mode either inside the cluster or from an external client: Cluster mode (default): Submitting Spark batch application and having the driver run on a host in your driver resource group. The spark-submit syntax is --deploy-mode cluster.
Click on the HDFS Web UI. A new web page is opened to show the Hadoop DFS (Distributed File System) health status. Click on the Spark Web UI. Another web page is opened showing the spark cluster and job status.
The Master Spark REST URL is on port 6066 by default. So you should see this as your Master endpoint: spark://172.20.9.151:6066.
If you go the to Spark web console (http://master:8080) you will get the details of the various endpoints of your cluster.
85
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
