Connect to Spark running on VM

Question

I have a Spark enviroment running on Ubuntu 16.2 over VirtualBox. Its configured to run locally and when I start Spark with

./start-all

I can access to it on VM via web-ui using the URL: http://localhost:8080

From the host machine (windows), I can access it too using the VM IP: http://192.168.x.x:8080.

The problem appears when I try to create a context from my host machine. I have a project in eclipse that uses maven, and I try to run the following code:

ConfigLoader.masterEndpoint = "spark://192.168.1.132:7077"

val conf = new SparkConf().setMaster(ConfigLoader.masterEndpoint).setAppName("SimpleApp")
val sc = new SparkContext(conf)

I got this error:

16/12/21 00:52:05 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://192.168.1.132:8080...
16/12/21 00:52:06 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master 192.168.1.132:8080
org.apache.spark.SparkException: Exception thrown in awaitResult
    at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)

I've tried changing the URL for:

ConfigLoader.masterEndpoint = "spark://192.168.1.132:7077"

Unsuccessfully.

Also, if I try to access directly to the master URL via web (http://localhost:7077 in VM), I don't get anything. I don't know if its normal.

What am I missing?

Answer 1

In your VM go to spark-2.0.2-bin-hadoop2.7/conf directory and create spark-env.sh file using below command.

cp spark-env.sh.template spark-env.sh

Open spark-env.sh file in vi editor and add below line.

SPARK_MASTER_HOST=192.168.1.132

Stop and start Spark using stop-all.sh and start-all.sh. Now in your program you can set the master like below.

val spark = SparkSession.builder()
  .appName("SparkSample")
  .master("spark://192.168.1.132:7077")
  .getOrCreate()