Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Active tasks is a negative number in Spark UI

Tags:

When using spark-1.6.2 and pyspark, I saw this:

enter image description here

where you see that the active tasks are a negative number (the difference of the the total tasks from the completed tasks).

What is the source of this error?


Node that I have many executors. However, it seems like there is a task that seems to have been idle (I don't see any progress), while another identical task completed normally.


Also this is related: that mail I can confirm that many tasks are being created, since I am using 1k or 2k executors.

The error I am getting is a bit different:

16/08/15 20:03:38 ERROR LiveListenerBus: Dropping SparkListenerEvent because no remaining room in event queue. This likely means one of the SparkListeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.
16/08/15 20:07:18 WARN TaskSetManager: Lost task 20652.0 in stage 4.0 (TID 116652, myfoo.com): FetchFailed(BlockManagerId(61, mybar.com, 7337), shuffleId=0, mapId=328, reduceId=20652, message=
org.apache.spark.shuffle.FetchFailedException: java.util.concurrent.TimeoutException: Timeout waiting for task.
like image 567
gsamaras Avatar asked Aug 15 '16 22:08

gsamaras


People also ask

How does Spark decide number of tasks?

Spark automatically sets the number of “map” tasks to run on each file according to its size (though you can control it through optional parameters to SparkContext. textFile , etc), and for distributed “reduce” operations, such as groupByKey and reduceByKey , it uses the largest parent RDD's number of partitions.

What is Spark UI task?

The number of tasks you could see in each stage is the number of partitions that spark is going to work on and each task inside a stage is the same work that will be done by spark but on a different partition of data.

What is executor and task in Spark?

Executors are worker nodes' processes in charge of running individual tasks in a given Spark job. They are launched at the beginning of a Spark application and typically run for the entire lifetime of an application. Once they have run the task they send the results to the driver.


1 Answers

It is a Spark issue. It occurs when executors restart after failures. The JIRA issue for the same is already created. You can get more details about the same from https://issues.apache.org/jira/browse/SPARK-10141 link.

like image 176
Hokam Avatar answered Oct 16 '22 00:10

Hokam