Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to add the "--deploy-mode cluster" option to my scala code

209/5000 Hello I want to add the option "--deploy-mode cluster" to my code scala:

  val sparkConf = new SparkConfig ().setMaster ("spark: //192.168.60.80:7077")

Without using the shell (the command. \ Spark-submit)

i whant to usage the " spark.submit.deployMode " in scala

like image 683
hatem dagbouj Avatar asked Jan 05 '23 01:01

hatem dagbouj


1 Answers

with SparkConfig:

//set up the spark configuration and create contexts
val sparkConf = new SparkConf().setAppName("SparkApp").setMaster("spark: //192.168.60.80:7077")

val sc = new SparkContext(sparkConf).set("spark.submit.deployMode", "cluster")

with SparkSession:

val spark = SparkSession
   .builder()
   .appName("SparkApp")
   .master("spark: //192.168.60.80:7077")
   .config("spark.submit.deployMode","cluster")
   .enableHiveSupport()
   .getOrCreate()
like image 182
FaigB Avatar answered Mar 04 '23 17:03

FaigB