Why do Spark jobs fail with org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 0 in speculation mode?

Question

I'm running a Spark job with in a speculation mode. I have around 500 tasks and around 500 files of 1 GB gz compressed. I keep getting in each job, for 1-2 tasks, the attached error where it reruns afterward dozens of times (preventing the job to complete).

org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 0

Any idea what is the meaning of the problem and how to overcome it?

org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output location for shuffle 0     at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$1.apply(MapOutputTracker.scala:384)     at org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$1.apply(MapOutputTracker.scala:381)     at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)     at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)     at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)     at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)     at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)     at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)     at org.apache.spark.MapOutputTracker$.org$apache$spark$MapOutputTracker$$convertMapStatuses(MapOutputTracker.scala:380)     at org.apache.spark.MapOutputTracker.getServerStatuses(MapOutputTracker.scala:176)     at org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$.fetch(BlockStoreShuffleFetcher.scala:42)     at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:40)     at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:92)     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)     at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)     at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)     at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)     at org.apache.spark.rdd.FlatMappedRDD.compute(FlatMappedRDD.scala:33)     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)     at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)     at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)     at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)     at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)     at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)     at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)     at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)     at org.apache.spark.scheduler.Task.run(Task.scala:56)     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)     at java.lang.Thread.run(Thread.java:722) 

Answer 1

This happened to me when I gave more memory to the worker node than it has. Since it didn't have swap, spark crashed while trying to store objects for shuffling with no more memory left.

Solution was to either add swap, or configure the worker/executor to use less memory in addition with using MEMORY_AND_DISK storage level for several persists.