This diagram is quite clear on the relationship between the different YARN and Spark memory related settings, except when it comes to spark.python.worker.memory
.
How does spark.python.worker.memory
fit into this memory model?
Are the Python processes governed by spark.executor.memory
or yarn.nodemanager.resource.memory-mb
?
Update
This question explains what the setting does, but doesn't answer the question concerning the memory governance, or how it relates to other memory settings.
Found this thread from the Apache-spark mailing list, and it appears that spark.python.worker.memory is a subset of the memory from spark.executor.memory.
From the thread: "spark.python.worker.memory is used for Python worker in executor"
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With