Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How does spark.python.worker.memory relate to spark.executor.memory?

This diagram is quite clear on the relationship between the different YARN and Spark memory related settings, except when it comes to spark.python.worker.memory.

How does spark.python.worker.memory fit into this memory model?

Are the Python processes governed by spark.executor.memory or yarn.nodemanager.resource.memory-mb?

Update

This question explains what the setting does, but doesn't answer the question concerning the memory governance, or how it relates to other memory settings.

like image 214
Douglas Willcocks Avatar asked Apr 13 '16 09:04

Douglas Willcocks


1 Answers

Found this thread from the Apache-spark mailing list, and it appears that spark.python.worker.memory is a subset of the memory from spark.executor.memory.

From the thread: "spark.python.worker.memory is used for Python worker in executor"

like image 190
David Avatar answered Sep 18 '22 06:09

David