What happens when an executor is lost?

Question

I get these messages:

16/05/22 13:33:53 ERROR YarnScheduler: Lost executor 61 on <host>: Executor heartbeat timed out after 134828 ms
16/05/22 13:33:53 WARN TaskSetManager: Lost task 25.0 in stage 12.0 (TID 2214, <host>): ExecutorLostFailure (executor 61 lost)

Will a replacement executor be spawned?

Answer 1

Will a replacement executor be spawned?

Yes, it will. Sparks DAGScheduler and it's lower level cluster manager implementation (Standalone, YARN or Mesos) will notice a task failed and will take care of rescheduling the said task as part of the overall stages executed.

DAGScheduler does three things in Spark (thorough explanations follow):

• Computes an execution DAG, i.e. DAG of stages, for a job.
• Determines the preferred locations to run each task on.
• Handles failures due to shuffle output files being lost.

For more on that you can find in Advanced Spark Tutorial and in Mastering Apache Spark.