Erlang reported epmd_closed error

Question

Erlang ver: R16B

OS: Fedora17

erl -name a and erl -sname a all reported following error:

{error_logger,{{2013,4,20},{14,50,20}},"Protocol: ~tp: register/listen error: ~tp~n",["inet_tcp",epmd_close]}
{error_logger,{{2013,4,20},{14,50,20}},crash_report,[[{initial_call,{net_kernel,init,['Argument__1']}},{pid,<0.21.0>},{registered_name,[]},{error_info,{exit,{error,badarg},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,320}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[net_sup,kernel_sup,<0.10.0>]},{messages,[]},{links,[#Port<0.56>,<0.18.0>]},{dictionary,[{longnames,true}]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,728}],[]]}
{error_logger,{{2013,4,20},{14,50,20}},supervisor_report,[{supervisor,{local,net_sup}},{errorContext,start_error},{reason,{'EXIT',nodistribution}},{offender,[{pid,undefined},{name,net_kernel},{mfargs,{net_kernel,start_link,[[s,longnames]]}},{restart_type,permanent},{shutdown,2000},{child_type,worker}]}]}
{error_logger,{{2013,4,20},{14,50,20}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}},{offender,[{pid,undefined},{name,net_sup},{mfargs,{erl_distribution,start_link,[]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}
{error_logger,{{2013,4,20},{14,50,20}},crash_report,[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.9.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[<0.8.0>]},{messages,[{'EXIT',<0.10.0>,normal}]},{links,[<0.8.0>,<0.7.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,117}],[]]}
{error_logger,{{2013,4,20},{14,50,20}},std_info,[{application,kernel},{exited,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{kernel,start,[normal,[]]}}}"}

Crash dump was written to: erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{k

Following is iptables:

> iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
DROP       icmp --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Following is erl:

> erl
Erlang R16B (erts-5.10.1) [source] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V5.10.1  (abort with ^G)
1>

It is on Linode VPS and used to be ok. I don't know what's the cause of this problem.

Answer 1

Probably, the problem is with your iptables configuration.

By default, Erlang doesn't allow connections to epmd from non-local addresses. If I'm not mistaken this behavior is actual in new Erlang versions. It seems like you have iptables issues related to NAT/MASQUERADING - your outgoing traffic is masqueraded to an external IP address by iptables (including traffic from/to 127.0.0.1).

Example

Masquerading all outgoing traffic

-A POSTROUTING -j SNAT --to-source x.x.x.x

This is output of 'iptables -t nat --list'

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
SNAT       all  --  anywhere             anywhere             to:x.x.x.x

Check 'erl -sname a' - doesn't work, exactly the same error

Now, masquerading all outgoing traffic except directed to 127.0.0.0/8

-A POSTROUTING ! -d 127.0.0.0/8 -j SNAT --to-source x.x.x.x

Pay attention to added '! -d 127.0.0.0/8'

Check iptables again

#iptables -t nat --list
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
SNAT       all  --  anywhere            !127.0.0.0/8          to:x.x.x.x

And check erlang again:

#erl -sname a
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9.1  (abort with ^G)
(a@akitaki)1> q().
ok
(a@akitaki)2> %

It works perfectly.

So, try to check your iptables/firewall NAT/MASQUERADING configuration.

UPDATE

To implement NAT/MASQUERADING you can use one of these iptables rules:

-A POSTROUTING ! -d 127.0.0.0/8 -j SNAT --to-source x.x.x.x

and

-A POSTROUTING ! -d 127.0.0.0/8 -j MASQUERADE

In the first case ('source NAT') you should point your external IP address explicitly. You can know out it using this command:

#ifconfig

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
    options=3<RXCSUM,TXCSUM>
    inet 127.0.0.1 netmask 0xff000000
en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    ether 70:56:81:98:d1:b9
    inet X.X.X.X netmask 0xffffff00 broadcast Y.Y.Y.Y
    media: autoselect
    status: active

inet X.X.X.X netmask 0xffffff00 broadcast Y.Y.Y.Y

here X.X.X.X is your external IP address you should point in '--SNAT x.x.x.x' command

In the second case ('masquerading') you don't point your external IP address explicitly - your server will do that automatically.

Answer 2

The problem has to do with epmd not being able to do its work. Everything else seem to be fallout from that problem. I would focus on epmd and get that up and working in the VPS first and then focus on the Erlang node.

For some reason, you don't get epmd up and then things fail all over the place as a result.