How to properly deploy with Akka Cluster

Question

Scenario

Right now we only have a single node running the whole system. What we want is to make a distinction between "frontend" nodes, and a single "backend" node.

• "Frontend" nodes (N nodes): Maintains a persistent connection with the clients through a WebSocket connection
• "Backend" node (1 node): Processes all the requests coming form all the frontend nodes querying to database, and handling the needed domain logic.

This distinction is needed due to some reasons:

• Do not reach the limit of 70-100k persistent connections per frontend node
• Avoid disconnecting the clients while deploying changes only affecting the backend

Work done

We have connected the actors living on the frontend node with the ones living on the backend. We've done so instantiating the backend node ActorRefs from the frontend using the akka.cluster.singleton.ClusterSingletonProxy, and the ClusterSingletonManager while really instantiating them in the backend.

Question

How we do the deploy taking into account the Akka cluster node downing notification?

As far as I understood by the Akka Cluster documentation about downing, and some comments on the akka mailing list, the recommended approach while dealing with that process would be something like:

1. Download the akka distribution from http://akka.io/downloads/
2. Copy and paste the akka-cluster bash script together with the jmxsh-R5.jar on a resources/bin/ folder (for instance)
3. Include that folder on the distributed package (I've added the following lines on the build.sbt): mappings in Universal ++= (baseDirectory.value / "resources" / "bin" * "*" get) map (bin => bin -> ("bin/" + bin.getName))
4. While deploying, set the node to be deployed as down manually calling the bash script like:
   • Execute bin/akka-cluster %node_to_be_deployed:port% down
   • Deploy the new code version
   • Execute bin/akka-cluster %deployed_node:port% join

Doubts:

1. Is this step by step procedure correct?
2. If the node to be deployed will have the very same IP and port after the deploy, is it needed to make the down and join?
3. We're planning to set one frontend and the backend nodes as the seed ones. This way, all the cluster could be reconstructed while making a deploy only of the frontend nodes, or only to the backend one. Is it correct?

Thanks!

Answer 1

To avoid downing manually, cleanup when a node is terminated, see: http://doc.akka.io/docs/akka/current/scala/cluster-usage.html#How_To_Cleanup_when_Member_is_Removed

Regarding your points:

1. You do not need this procedure when the JVM is restarted and the cleanup code is executed. Only when the cleanup code somehow failed, you need to down manually as described in the procedure.
2. When the node is marked as removed by other nodes (after the cleanup code is executed), the same ip and port combination can be used to re-join the cluster.
3. Yes, you can just re-deploy a frontend node.

PS.:
- Coordinated shutdown will be improved in akka 2.5, see: https://github.com/akka/akka-meta/issues/38
- If you want to manage your cluster using a http API, see: http://developer.lightbend.com/docs/akka-cluster-management/current/