Lots of ERROR ErrorMonitor: AssociationError on spark startup

Question

I am using spark on mesos (with kafka and cassandra) On startup, I have a lot of errors (~ 100) then evrything works fine. The errors are of this type:

[Stage 0:=======> (24 + 26) / 50][Stage 3:> (0 + 24) / 24]15/09/17 09:48:35 ERROR ErrorMonitor: AssociationError [akka.tcp://[email protected]:58325] <- [akka.tcp://[email protected]:59441]: Error [Shut down address: akka.tcp://[email protected]:59441] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://[email protected]:59441 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ]

What could be the reason of this error? How can I solve it?

Answer 1

The log seems to be noise.

This issue also happens at Spark 1.5.0 in CDH 5.5.0, and Cloudera says below.

http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_rn_spark_ki.html

When using Spark on YARN, the driver reports misleading error messages The Spark driver reports misleading error messages such as:

ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@...] -> [akka.tcp://sparkExecutor@...]: Error [Association failed with [akka.tcp://sparkE xecutor@...]] [akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor@...]]

Workaround: Add the following property to the Spark log4j configuration file: log4j.logger.org.apache.spark.rpc.akka.ErrorMonitor=FATAL. See Configuring Spark Application Logging Properties.