RabbitMQ Generic server rabbit_disk_monitor terminating / eheap_alloc: Cannot allocate 229520 bytes of memory (of type "old_heap")

Question

RabbitMQ crashed. RabbitMQ was working correctly for many days(10-15 days). I am not getting why it got crashed.

---

I am using RabbitMQ 3.4.0 on Erlang 17.0

---

The erlang has created dump file for the crash. Which shows

eheap_alloc: Cannot allocate 229520 bytes of memory (of type "old_heap").

---

Also note that the rabbitmq publish-subscribe message load is very low. (max:1-2 messages/second).And RabbitMQ messages are processed as it comes so RabbitMQ is almost empty all the time. The disk space & memory are also sufficient.

More system info:
Limiting to approx 8092 file handles (7280 sockets)
Memory limit set to 6553MB of 16383MB total.
Disk free limit set to 50MB.

The RabbitMQ logs are as below.

=ERROR REPORT==== 18-Jul-2015::04:29:31 ===
** Generic server rabbit_disk_monitor terminating 
** Last message in was update
** When Server state == {state,"c:/Users/jasmin.joshi/AppData/Roaming/RabbitMQ/db/rabbit@localhost-mnesia",
                               50000000,28358258688,100,10000,
                               #Ref<0.0.106.70488>,false}
** Reason for termination == 
** {eacces,[{erlang,open_port,
                    [{spawn,"C:\\Windows\\system32\\cmd.exe /c dir /-C /W \"c:/Users/jasmin.joshi/AppData/Roaming/RabbitMQ/db/rabbit@localhost-mnesia\""},
                     [stream,in,eof,hide]],
                    []},
            {os,cmd,1,[{file,"os.erl"},{line,204}]},
            {rabbit_disk_monitor,get_disk_free,2,[]},
            {rabbit_disk_monitor,internal_update,1,[]},
            {rabbit_disk_monitor,handle_info,2,[]},
            {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,599}]},
            {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

=INFO REPORT==== 18-Jul-2015::04:29:31 ===
Disabling disk free space monitoring on unsupported platform:
{{'EXIT',{eacces,[{erlang,open_port,
                          [{spawn,"C:\\Windows\\system32\\cmd.exe /c dir /-C /W \"c:/Users/jasmin.joshi/AppData/Roaming/RabbitMQ/db/rabbit@localhost-mnesia\""},
                           [stream,in,eof,hide]],
                          []},
                  {os,cmd,1,[{file,"os.erl"},{line,204}]},
                  {rabbit_disk_monitor,get_disk_free,2,[]},
                  {rabbit_disk_monitor,init,1,[]},
                  {gen_server,init_it,6,[{file,"gen_server.erl"},{line,306}]},
                  {proc_lib,init_p_do_apply,3,
                            [{file,"proc_lib.erl"},{line,239}]}]}},
 17179336704}

=INFO REPORT==== 18-Jul-2015::04:29:31 ===
Disabling disk free space monitoring on unsupported platform:
{{'EXIT',{eacces,[{erlang,open_port,
                          [{spawn,"C:\\Windows\\system32\\cmd.exe /c dir /-C /W \"c:/Users/jasmin.joshi/AppData/Roaming/RabbitMQ/db/rabbit@localhost-mnesia\""},
                           [stream,in,eof,hide]],
                          []},
                  {os,cmd,1,[{file,"os.erl"},{line,204}]},
                  {rabbit_disk_monitor,get_disk_free,2,[]},
                  {rabbit_disk_monitor,init,1,[]},
                  {gen_server,init_it,6,[{file,"gen_server.erl"},{line,306}]},
                  {proc_lib,init_p_do_apply,3,
                            [{file,"proc_lib.erl"},{line,239}]}]}},
 17179336704}

---

=CRASH REPORT==== 18-Jul-2015::04:29:31 ===
  crasher:
    initial call: rabbit_disk_monitor:init/1
    pid: <0.167.0>
    registered_name: rabbit_disk_monitor
    exception exit: {eacces,
                        [{erlang,open_port,
                             [{spawn,
                                  "C:\\Windows\\system32\\cmd.exe /c dir /-C /W \"c:/Users/jasmin.joshi/AppData/Roaming/RabbitMQ/db/rabbit@localhost-mnesia\""},
                              [stream,in,eof,hide]],
                             []},
                         {os,cmd,1,[{file,"os.erl"},{line,204}]},
                         {rabbit_disk_monitor,get_disk_free,2,[]},
                         {rabbit_disk_monitor,internal_update,1,[]},
                         {rabbit_disk_monitor,handle_info,2,[]},
                         {gen_server,handle_msg,5,
                             [{file,"gen_server.erl"},{line,599}]},
                         {proc_lib,init_p_do_apply,3,
                             [{file,"proc_lib.erl"},{line,239}]}]}
      in function  gen_server:terminate/6 (gen_server.erl, line 746)
    ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.140.0>]
    messages: []
    links: [<0.166.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 4185
    stack_size: 27
    reductions: 481081978
  neighbours:

=SUPERVISOR REPORT==== 18-Jul-2015::04:29:31 ===
     Supervisor: {local,rabbit_disk_monitor_sup}
     Context:    child_terminated
     Reason:     {eacces,
                     [{erlang,open_port,
                          [{spawn,
                               "C:\\Windows\\system32\\cmd.exe /c dir /-C /W \"c:/Users/jasmin.joshi/AppData/Roaming/RabbitMQ/db/rabbit@localhost-mnesia\""},
                           [stream,in,eof,hide]],
                          []},
                      {os,cmd,1,[{file,"os.erl"},{line,204}]},
                      {rabbit_disk_monitor,get_disk_free,2,[]},
                      {rabbit_disk_monitor,internal_update,1,[]},
                      {rabbit_disk_monitor,handle_info,2,[]},
                      {gen_server,handle_msg,5,
                          [{file,"gen_server.erl"},{line,599}]},
                      {proc_lib,init_p_do_apply,3,
                          [{file,"proc_lib.erl"},{line,239}]}]}
     Offender:   [{pid,<0.167.0>},
                  {name,rabbit_disk_monitor},
                  {mfargs,{rabbit_disk_monitor,start_link,[50000000]}},
                  {restart_type,{transient,1}},
                  {shutdown,4294967295},
                  {child_type,worker}]


=CRASH REPORT==== 18-Jul-2015::04:29:31 ===
  crasher:
    initial call: rabbit_disk_monitor:init/1
    pid: <0.24989.51>
    registered_name: []
    exception exit: unsupported_platform
      in function  gen_server:init_it/6 (gen_server.erl, line 322)
    ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.140.0>]
    messages: []
    links: [<0.166.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 650
  neighbours:

=SUPERVISOR REPORT==== 18-Jul-2015::04:29:31 ===
     Supervisor: {local,rabbit_disk_monitor_sup}
     Context:    start_error
     Reason:     unsupported_platform
     Offender:   [{pid,<0.167.0>},
                  {name,rabbit_disk_monitor},
                  {mfargs,{rabbit_disk_monitor,start_link,[50000000]}},
                  {restart_type,{transient,1}},
                  {shutdown,4294967295},
                  {child_type,worker}]


=CRASH REPORT==== 18-Jul-2015::04:29:31 ===
  crasher:
    initial call: rabbit_disk_monitor:init/1
    pid: <0.24991.51>
    registered_name: []
    exception exit: unsupported_platform
      in function  gen_server:init_it/6 (gen_server.erl, line 322)
    ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.140.0>]
    messages: []
    links: [<0.166.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 650
  neighbours:

=SUPERVISOR REPORT==== 18-Jul-2015::04:29:31 ===
     Supervisor: {local,rabbit_disk_monitor_sup}
     Context:    start_error
     Reason:     unsupported_platform
     Offender:   [{pid,{restarting,<0.167.0>}},
                  {name,rabbit_disk_monitor},
                  {mfargs,{rabbit_disk_monitor,start_link,[50000000]}},
                  {restart_type,{transient,1}},
                  {shutdown,4294967295},
                  {child_type,worker}]

Answer 1

From the error message, rabbitmq can't open more files due to system limits. You can set max open file numbers to upper value to avoid the problem.

https://serverfault.com/questions/249477/windows-server-2008-r2-max-open-files-limit