Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to pass external parameters through Spark submit

In my Application, i need to connect to the database so i need to pass IP address and database name when application is submitted.

I submit the application as follows: :

./spark-submit --class class name --master spark://localhost:7077 \
--deploy-mode client /home/hadoop/myjar.jar
like image 932
ROOT Avatar asked Mar 03 '16 21:03

ROOT


People also ask

What happens when spark job is submitted?

What happens when a Spark Job is submitted? When a client submits a spark user application code, the driver implicitly converts the code containing transformations and actions into a logical directed acyclic graph (DAG).

Can you explain what happens internally when we submit a spark job using spark submit?

The entire resource allocation and the tracking of the jobs and tasks are performed by the cluster manager. As soon as you do a Spark submit, your user program and other configuration mentioned are copied onto all the available nodes in the cluster. So that the program becomes the local read on all the worker nodes.


1 Answers

As stated by zero323 you can use the spark-submit command from the link

  ./bin/spark-submit \
  --class <main-class>
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

Here, conf is used to pass the Spark related configs which are required for the application to run like any specific property(executor memory) or if you want to override the default property which is set in Spark-default.conf.

As far as your use case is concerned you want to pass the IP to the application to connect to database then you can use the [application-arguments] which are passed after the JAR.

When you set up your main as:

def main(args: Array[String])

Then you can accept anything as an argument given after .jar line.

Please refer for more details

like image 55
Ajit K'sagar Avatar answered Oct 25 '22 08:10

Ajit K'sagar