I am running a spark cluster over C++ code wrapped in python. I am currently testing different configurations of multi-threading options (at Python level or Spark level).
I am using spark with standalone binaries, over a HDFS 2.5.4 cluster. The cluster is currently made of 10 slaves, with 4 cores each.
From what I can see, by default, Spark launches 4 slaves per node (I have 4 python working on a slave node at a time).
How can I limit this number ? I can see that I have a --total-executor-cores option for "spark-submit", but there is little documentation on how it impacts the distribution of executors over the cluster !
I will run tests to get a clear idea, but if someone knowledgeable has a clue of what this option does, it could help.
Update :
I went through spark documentation again, here is what I understand :
--total-executor-cores
whith spark-submit
(least satisfactory, since there is no clue on how the pool of cores is dealt with)SPARK_WORKER_CORES
in the configuration file-c
options with the starting scripts The following lines of this documentation http://spark.apache.org/docs/latest/spark-standalone.html helped me to figure out what is going on :
SPARK_WORKER_INSTANCES
Number of worker instances to run on each machine (default: 1). You can make this more than 1 if you have have very large machines and would like multiple Spark worker processes. If you do set this, make sure to also set SPARK_WORKER_CORES explicitly to limit the cores per worker, or else each worker will try to use all the cores.
What is still unclear to me is why it is better in my case to limit the number of parallel tasks per worker node to 1 and rely on my C++ legacy code multithreading. I will update this post with experiment results, when I will finish my study.
The consensus in most Spark tuning guides is that 5 cores per executor is the optimum number of cores in terms of parallel processing.
According to the recommendations which we discussed above: Leave 1 core per node for Hadoop/Yarn daemons => Num cores available per node = 16-1 = 15. So, Total available of cores in cluster = 15 x 10 = 150. Number of available executors = (total cores/num-cores-per-executor) = 150/5 = 30.
Once you do a Spark submit, a driver program is launched and this requests for resources to the cluster manager and at the same time the main program of the user function of the user processing program is initiated by the driver program.
The number of cores can be specified with the --executor-cores flag when invoking spark-submit, spark-shell, and pyspark from the command line, or by setting the spark. executor. cores property in the spark-defaults. conf file or on a SparkConf object.
The documentation does not seem clear.
From my experience, the most common practice to allocate resources is by indicating the number of executors and the number of cores per executor, for example (taken from here):
$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi \
--master yarn-cluster \
--num-executors 10 \
--driver-memory 4g \
--executor-memory 2g \
--executor-cores 4 \
--queue thequeue \
lib/spark-examples*.jar \
10
However, this approach is limited to YARN, and is not applicable to standalone and mesos based Spark, according to this.
Instead, the parameter --total-executor-cores
can be used, which represents the total amount of cores - of all executors - assigned to the Spark job. In your case, having a total of 40 cores, setting the attribute --total-executor-cores 40
would make use of all the available resources.
Unfortunately, I am not aware of how Spark distributes the workload when less resources than the total available are provided. If working with two or more simultaneous jobs, however, it should be transparent to the user, in that Spark (or whatever resource manager) would manage how the resources are managed depending on the user settings.
To make sure how many workers started on each slave, open web browser, type http://master-ip:8080, and see the workers
section about how many workers has been started exactly, and also which worker on which slave. (I mention these above because I am not sure what do you mean by saying '4 slaves per node')
By default, spark would start exact 1 worker on each slave unless you specify
SPARK_WORKER_INSTANCES=n
in conf/spark-env.sh, where n is the number of worker instance you would like to start on each slave.
When you submit a spark job through spark-submit, spark would start an application driver and several executors for your job.
--total-executor-cores
you specified would limit the total cores that is available to this application.If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With