What is the difference between RUNNING and LOADING states for executors in web UI?

Question

We are currently using Spark 1.5.2 and have a standalone cluster with three nodes (1 master, 2 workers) and we are running a streaming job on the cluster. There are no problems with the streaming job itself and we verified by logs and performance metrics (like looking at the CPU usage, disk usage, memory usage) that both workers are being used.

Our streaming job currently reads from a Cassandra cluster and stores that information to Elasticsearch. The other thing to note is that there is only one receiver for the streaming job, so that means only one worker will be reading from Cassandra (both workers will however write to Elasticsearch).

When we go into the application view (by clicking on the streaming job application ID listed in the running applications table from the master node) from the Spark web UI, there is an Executor Summary table with a few columns showing the workers, cores, memory, state and logs.

For our streaming job, it normally looks as follows:

Note that the state shown here are both RUNNING.

When we restart the streaming job, one of the states for one of the worker would sometimes be in the LOADING state as follows:

Once it is shown to be in the LOADING state, it will remain in the state (i.e. does not eventually change into running state).

So the question is what is the difference between the RUNNING and LOADING states shown in the Spark web UI?

In terms of functionality, both streaming jobs seem to do what we expect it to do. In other words, regardless of the state, the Spark worker seems to be doing work and there does not seem to be any obvious problems (not in the logs anyway).

Answer 1

A LOADING Executor is on the way to RUNNING, but hasn't yet been registered with the Master, so it isn't quite ready to do useful work.

Answer 2

After much thinking about potential scenario I couldn't figure out one, but I think the only way to hit the issue is when you somehow lost some updates from executors.

It doesn't really matter since web UI is just a SparkListener that receives updates from the infrastructure and displays it in a nice visual way that you can access using your browser. Whether it has received all the events doesn't prevent the infrastructure (like executors) working properly, and it appears it does not.

It can be annoying, though, and if reproducible, I'd file an issue since it reduces the point of web UI for monitoring.

p.s. You can register your own SparkListener and do a "web UI" yourself. If you do, you could see the events coming in and compare with the real web UI.