How to add seed nodes to a running akka-cluster

Question

I am trying to create a fault tolerant akka-cluster which will be deployed in AWS. Its a standard cluster with 3 seed-nodes SN1, SN2 and SN3 and multiple Akka-systems connected to one seed-node. The SN's have ASG's which will spin a new instance in case one goes down.

I know that if SN1 fails the system will failover to the other SN (seed-nodes). But how can I register SN1 which now has a new IP address seamlessly without stopping the cluster.

I tried the following with no luck

• Cluster.joinseednodes only works on startup
• ELB behind each SN did not work. Akka couldn't connect to the ELB

Does anyone have ideas as to how one can add seed-nodes to a running Akka-cluster?

Thanks

Answer 1

Please check this:

http://chrisloy.net/2014/05/11/akka-cluster-ec2-autoscaling.html

Is in scala, but is using autoscalling too and explains about how to find your new IP (S1) and the rest of ips (S2...) and joining them.

Hope will help.

Answer 2

We ended up registering the ip address:port to zookeeper via a zookeeper-seed plugin. The plugin implements a latch if I am not mistaken.

Basically Each aka system gets the list of systems from zookeeper. It then removes itself from the list and uses the new list as its seed nodes.

If I get the time I will try to write up something on the lessons learned implementing a 24 system akka cluster.

Thanks