How to tune memory for Spark Application running in local mode

Question

Running Spark Application with local mode, I used the command, such as:

spark-submit --master local[*] my_spark_application.py

In this case, dose which mean that my application used all memory of my local computer? Are the other parameters, like driver-memory and executor-memory still working?

Answer 1

Setting driver memory is the only way to increase memory in a local spark application.

"Since you are running Spark in local mode, setting spark.executor.memory won't have any effect, as you have noticed. The reason for this is that the Worker "lives" within the driver JVM process that you start when you start spark-shell and the default memory used for that is 512M. You can increase that by setting spark.driver.memory to something higher, for example 5g" from How to set Apache Spark Executor memory

Answer 2

It depends on which virtual environment tool is used , if you install just spark without virtual environment (like docker) it takes your full your local memory , So , I recommend to use spark inside docker container which it takes about 220MB (default)

First install docker ;

then , Create container ;

install spark into container .