Starting remote Erlang nodes

Question

I want to write a master-slave application in Erlang. I am thinking at the following things I need from the architecture:

• the slaves shouldn't die when the master dies, but rather try to reconnect to it while the master is down

• the master should automatically start the remote nodes if they don't connect automatically or they are down (probably the supervisor behaviour in OTP)

Is there a OTP oriented behaviour to do this? I know I can start remote nodes with slave:start_link() and I can monitor nodes with erlang:monitor(), but I don't know how this can be incorporated in a gen_server behaviour.

Answer 1

I agree with the comments about using erlang:monitor_node and the use of distributed applications.

You cannot just use the slave module to accomplish that, it clearly states "All slave nodes which are started by a master will terminate automatically when the master terminates".

There is currently no OTP behaviour to do it either. Supervision trees are hierarchical ; it seems like you are looking for something where there is a hierarchy in terms of application logic, but spawning is done an a peer-to-peer basis (or an individual basis, depending upon your point of view).

If you were to use multiple Erlang VMs then you should carefully consider how many you run, as a large number of them may cause performance issues due to the OS swapping OS processes in and out. A rule of thumb for best performance is to aim for having no more than one OS process (i.e. one Erlang VM) per CPU core.

Answer 2

If you're interested in studying other implementations, Basho's riak_core framework has a pretty good take on decentralized distributed applications.

riak_core_node_watcher.erl has most of the interesting node observation code in it.

Search and you'll find there are quite a few talks and presentations about the framework.