Spark - Remote Akka Client Disassociated

Question

I am setting up Spark 0.9 on AWS and am finding that when launching the interactive Pyspark shell, my executors / remote workers are first being registered:

14/07/08 22:48:05 INFO cluster.SparkDeploySchedulerBackend: Registered executor:
Actor[akka.tcp://[email protected]:54110/user/
Executor#-862786598] with ID 0

and then disassociated almost immediately, before I have the chance to run anything:

14/07/08 22:48:05 INFO cluster.SparkDeploySchedulerBackend: Executor 0 disconnected, 
so removing it
14/07/08 22:48:05 ERROR scheduler.TaskSchedulerImpl: Lost an executor 0 (already 
removed): remote Akka client disassociated

Any idea what might be wrong? I've tried adjusting the JVM options spark.akka.frameSize and spark.akka.timeout, but I'm pretty sure this is not the issue since (1) I'm not running anything to begin with, and (2) my executors are disconnecting a few seconds after startup, which is well within the default 100s timeout.

Thanks!

Jack

Answer 1

I had a very similar problem, if not the same. It started to work for me once the workers were connecting to master by using the very same name as the master thought it had.

My log messages were something like:

ERROR remote.EndpointWriter: AssociationError [akka.tcp://[email protected]:7078] -> [akka.tcp://[email protected]:7077]: Error [Association failed with [akka.tcp://[email protected]:7077]]. ERROR remote.EndpointWriter: AssociationError [akka.tcp://[email protected]:7078] -> [akka.tcp://[email protected]:7077]: Error [Association failed with [akka.tcp://[email protected]:7077]]

WARN util.Utils: Your hostname, idc1-hrm1 resolves to a loopback address: 127.0.0.1; using 192.168.121.187 instead (on interface eth0)

So check the log of the master and see what name it thinks it has. Then use that very same name on the workers.