Menu
I get these messages:
16/05/22 13:33:53 ERROR YarnScheduler: Lost executor 61 on <host>: Executor heartbeat timed out after 134828 ms
16/05/22 13:33:53 WARN TaskSetManager: Lost task 25.0 in stage 12.0 (TID 2214, <host>): ExecutorLostFailure (executor 61 lost)
Will a replacement executor be spawned?
870
Will a replacement executor be spawned? Yes, it will. Sparks DAGScheduler and it's lower level cluster manager implementation (Standalone, YARN or Mesos) will notice a task failed and will take care of rescheduling the said task as part of the overall stages executed.
spark.yarn.executor.memoryOverhead. Is just the max value .The goal is to calculate OVERHEAD as a percentage of real executor memory, as used by RDDs and DataFrames --executor-memory/spark.executor.memory.
Will a replacement executor be spawned?
Yes, it will. Sparks DAGScheduler and it's lower level cluster manager implementation (Standalone, YARN or Mesos) will notice a task failed and will take care of rescheduling the said task as part of the overall stages executed.
DAGScheduler does three things in Spark (thorough explanations follow):
- Computes an execution DAG, i.e. DAG of stages, for a job.
- Determines the preferred locations to run each task on.
- Handles failures due to shuffle output files being lost.
For more on that you can find in Advanced Spark Tutorial and in Mastering Apache Spark.
118
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
