Why are Apache Spark worker executor killed with exit status 1?

Question

All the workers in the apache spark standalone setup are showing the following message. Worker: Executor app-20150902151146-0001/6 finished with state KILLED exitStatus 1

what is the reason to get this message.

Answer 1

With the few information you are giving, we can only answer this question broadly.

So my answer is inspired from Sean Owen's answer concerning the relationship between workers and executors and also from the Cloudera Blog on Resource Management with YARN along with official documentation on the Cluster mode overview.

So let's begin with defining Apache Spark application architecture :

Spark Application Architecture

For those who are familiar with the Apache Spark API, an application corresponds to an instance of the SparkContext class. An application can be used for a single batch job, an interactive session with multiple jobs spaced apart, or a long-lived server continually satisfying requests. Unlike MapReduce, an application will have processes, called Executors, running on the cluster on its behalf even when it’s not running any jobs.

This approach enables data storage in memory for quick access, as well as lightning-fast task startup time.

Executors :

MapReduce runs each task in its own process. When a task completes, the process goes away. In Spark, many tasks can run concurrently in a single process, and this process sticks around for the lifetime of the Spark application, even when no jobs are running.

The advantage of this model, as mentioned above, is speed: Tasks can start up very quickly and process in-memory data. The disadvantage is coarser-grained resource management. As the number of executors for an app is fixed and each executor has a fixed allotment of resources, an app takes up the same amount of resources for the full duration that it’s running. (When YARN supports container resizing, we plan to take advantage of it in Spark to acquire and give back resources dynamically.)

So now that we have defined what is an Executor, let's define the Executor states

ExecutorState :

An executor can be in one of the following states :

LAUNCHING, LOADING, RUNNING, KILLED, FAILED, LOST, EXITED

The last four states describe a finished job, and for one of many reason, the following error :

Worker: Executor app-20150902151146-0001/6 finished with state KILLED exitStatus 1

means that your `app-20150902151146-0001̀ has been killed by an the Worker who stopped and asked to kill the executor. (code Ref.)