Could not parse Master URL: 'spark:http://localhost:18080'

Question

When I'm trying to run my code it throws this Exception:

Exception in thread "main" org.apache.spark.SparkException: Could not parse Master URL:spark:http://localhost:18080

This is my code:

SparkConf conf = new SparkConf().setAppName("App_Name").setMaster("spark:http://localhost:18080").set("spark.ui.port","18080");
JavaStreamingContext ssc = new JavaStreamingContext(sc, new Duration(1000));
String[] filet=new String[]{"Obama","ISI"};

JavaReceiverInputDStream<Status> reciverStream=TwitterUtils.createStream(ssc,filet);
JavaDStream<String> statuses = reciverStream.map(new Function<Status, String>() {
     public String call(Status status) { return status.getText(); }
     }
      );
ssc.start();
ssc.awaitTermination();}}

Any idea how can I fix this problem?

Answer 1

The problem is that you specify 2 schemas in the URL you pass to SparkConf.setMaster().

The spark is the schema, so you don't need to add http after spark. See the javadoc of SparkConf.setMaster() for more examples.

So the master URL you should be using is "spark://localhost:18080". Change this line:

SparkConf conf = new SparkConf().setAppName("App_Name")
    .setMaster("spark://localhost:18080").set("spark.ui.port","18080");