How to set YARN queue for spark-shell?

Question

I'm executing some spark(scala) sql code in spark shell. I want to know which queue I am using and if possible I want to know how much memory and executors I am using and how to optimize it?

Answer 1

You can set queue name, number of executors, executor memory, number of total cores, cores per executor, driver memory,etc when you start spark shell or spark-submit

here is how you can specify the parameters.

spark-shell --executor-memory 6G --executor-cores 5 --num-executors 20 --driver-memory 2G --queue $queue_name

You should be calculating these parameters as per your cluster capacity according to fat executor or thin executor concept.

If you still want to check resources utilization, you can check resource manager page or SPARK web UI page