Spark - How to run a standalone cluster locally

Question

Is there the possibility to run the Spark standalone cluster locally on just one machine (which is basically different from just developing jobs locally (i.e., local[*]))?.

So far I am running 2 different VMs to build a cluster, what if I could run a standalone cluster on the very same machine, having for instance three different JVMs running?

Could something like having multiple loopback addresses do the trick?

Answer 1

yes you can do it, launch one master and one worker node and you are good to go

launch master

./sbin/start-master.sh

launch worker

./bin/spark-class org.apache.spark.deploy.worker.Worker  spark://localhost:7077 -c 1 -m 512M

run SparkPi example

./bin/spark-submit  --class org.apache.spark.examples.SparkPi   --master spark://localhost:7077  lib/spark-examples-1.2.1-hadoop2.4.0.jar 

Apache Spark Standalone Mode Documentation