Does master node execute actual tasks in Spark?

Question

My question may sound silly, but it bothers me for a long time.

The picture shown above is the components of a distributed Spark application. I think this picture indicates that the master node will never execute actual tasks, but only is served as a cluster manager. Is it true?

By the way, the tasks here refers to the user-submit tasks.

Answer 1

Yes, the master node executes the driver process and does not run tasks. Tasks run in executor processes on the worker nodes. The master node is rarely stressed from a CPU standpoint but, depending on how broadcast variables, accumulators and collect are used, it may be quite stressed in terms of RAM usage.