Menu
I am trying to tune the memory parameter of Spark. I tried:
sparkSession.conf.set("spark.memory.storageFraction","0.1") //sparkSession has been created
After I submit the job and checked Spark UI. I found "Storage Memory" is still as before. So the above did not work.
What is the correct way to set "spark.memory.storageFraction"?
I am using Spark 2.0.
816
Earlier memory management was done using StaticMemoryManager class, in spark 1.6 and above, memory management is done by UnifiedMemoryManager class. For the sake of understanding, we will take an example of 4GB Memory allocated to an executor and leave the default configuration and see how much memory each segment gets.
Initial Storage Memory (50% of spark memory) — 1423MB — 34.75% The percentage value is only for 4GB executor memory calculation, for a different executor memory configuration, these won’t hold good. It is only given for understanding purposes.
This memory stores sparks internal objects. One thing to note is that, if the executor memory is less than 1.5 times of reserved memory, Spark will fail with a “please use larger heap size” error message. 2. User Memory
Hence, storage blocks can occupy parts of execution memory if it is free and vice-versa. The parameter spark.memory.fraction determines the total memory dedicated to Spark (for both shuffle and storage). The amount of storage memory which is protected from eviction is governed by spark.memory.storageFraction.
I face same problem , after read some code from spark github I think the "Storage Memory" on spark ui is misleading, it's not indicate the size of storage region，actually it represent the maxMemory:
maxMemory = (executorMemory - reservedMemory[default 384]) * memoryFraction[default 0.6]
check these for more detail ↓↓↓
spark ui executors-page source code
getMaxmemory source code
175
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
