RabbitMQ virtual host error when starting service

Question

I've had a RabbitMQ server running for months. This morning I was unable to connect to it, my applications was timing out and the Management client was unresponsive. Rebooted the machine. Applications are still timing out. I'm able to login to the Management client but I see this message:

Virtual host / experienced an error on node rabbit@MQT01 and may be inaccessible

All my queues are there but can't see any exchanges.

I hope someone can help me figure out what going on. I've looked at the logs but can't find any good hint.

Here a part of the log:

2018-09-11 09:39:42 =ERROR REPORT====
** Generic server <0.281.0> terminating
** Last message in was {'$gen_cast',{submit_async,#Fun<rabbit_queue_index.36.122888644>}}
** When Server state == undefined
** Reason for termination == 
** {function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]}
2018-09-11 09:39:42 =CRASH REPORT====
  crasher:
    initial call: worker_pool_worker:init/1
    pid: <0.281.0>
    registered_name: []
    exception exit: {{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]},[{gen_server2,terminate,3,[{file,"src/gen_server2.erl"},{line,1161}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
    ancestors: [worker_pool_sup,rabbit_sup,<0.262.0>]
    message_queue_len: 0
    messages: []
    links: [<0.276.0>,<0.336.0>,#Port<0.31196>]
    dictionary: [{fhc_age_tree,{1,{{10352640,#Ref<0.1077581647.1695285251.67028>},true,nil,nil}}},{worker_pool_worker,true},{rand_seed,{#{jump => #Fun<rand.16.15449617>,max => 288230376151711743,next => #Fun<rand.15.15449617>,type => exsplus},[257570830250844431|246837015578235662]}},{worker_pool_name,worker_pool},{{"c:/Users/dfpsb/AppData/Roaming/RabbitMQ/db/RABBIT~1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/9GD33C2I2PKZ7A8QHZ4MWWCKE/journal.jif",fhc_file},{file,1,true}},{{#Ref<0.1077581647.1695285251.67028>,fhc_handle},{handle,{file_descriptor,prim_file,{#Port<0.31196>,1808}},#Ref<0.1077581647.1695285251.67028>,240,false,0,infinity,[],<<>>,0,0,0,0,0,false,"c:/Users/dfpsb/AppData/Roaming/RabbitMQ/db/RABBIT~1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/queues/9GD33C2I2PKZ7A8QHZ4MWWCKE/journal.jif",[write,binary,raw,read],[{write_buffer,infinity}],true,true,10352640}}]
    trap_exit: false
    status: running
    heap_size: 10958
    stack_size: 27
    reductions: 104391
  neighbours:
    neighbour: [{pid,<0.279.0>},{registered_name,[]},{initial_call,{worker_pool_worker,init,['Argument__1']}},{current_function,{gen,do_call,4}},{ancestors,[worker_pool_sup,rabbit_sup,<0.262.0>]},{message_queue_len,0},{links,[<0.276.0>,<0.336.0>]},{trap_exit,false},{status,waiting},{heap_size,4185},{stack_size,42},{reductions,21548},{current_stacktrace,[{gen,do_call,4,[{file,"gen.erl"},{line,169}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,210}]},{file,call,2,[{file,"file.erl"},{line,1499}]},{rabbit_queue_index,get_journal_handle,1,[{file,"src/rabbit_queue_index.erl"},{line,881}]},{rabbit_queue_index,load_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,894}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,904}]},{rabbit_queue_index,scan_queue_segments,3,[{file,"src/rabbit_queue_index.erl"},{line,724}]},{rabbit_queue_index,queue_index_walker_reader,2,[{file,"src/rabbit_queue_index.erl"},{line,712}]}]}]
    neighbour: [{pid,<0.278.0>},{registered_name,[]},{initial_call,{worker_pool_worker,init,['Argument__1']}},{current_function,{gen,do_call,4}},{ancestors,[worker_pool_sup,rabbit_sup,<0.262.0>]},{message_queue_len,0},{links,[<0.276.0>,<0.336.0>,#Port<0.31157>]},{trap_exit,false},{status,waiting},{heap_size,6772},{stack_size,102},{reductions,129623},{current_stacktrace,[{gen,do_call,4,[{file,"gen.erl"},{line,169}]},{gen_server2,call,3,[{file,"src/gen_server2.erl"},{line,323}]},{array,sparse_foldr_3,6,[{file,"array.erl"},{line,1848}]},{array,sparse_foldr_2,8,[{file,"array.erl"},{line,1837}]},{lists,foldr,3,[{file,"lists.erl"},{line,1276}]},{rabbit_queue_index,scan_queue_segments,3,[{file,"src/rabbit_queue_index.erl"},{line,725}]},{rabbit_queue_index,queue_index_walker_reader,2,[{file,"src/rabbit_queue_index.erl"},{line,712}]},{rabbit_queue_index,'-queue_index_walker/1-fun-0-',2,[{file,"src/rabbit_queue_index.erl"},{line,694}]}]}]
    neighbour: [{pid,<0.280.0>},{registered_name,[]},{initial_call,{worker_pool_worker,init,['Argument__1']}},{current_function,{array,set_1,4}},{ancestors,[worker_pool_sup,rabbit_sup,<0.262.0>]},{message_queue_len,0},{links,[<0.276.0>,<0.336.0>,#Port<0.31170>]},{trap_exit,false},{status,runnable},{heap_size,121536},{stack_size,44},{reductions,122988},{current_stacktrace,[{array,set_1,4,[{file,"array.erl"},{line,590}]},{array,set_1,4,[{file,"array.erl"},{line,592}]},{array,set_1,4,[{file,"array.erl"},{line,592}]},{array,set,3,[{file,"array.erl"},{line,574}]},{rabbit_queue_index,parse_segment_publish_entry,5,[{file,"src/rabbit_queue_index.erl"},{line,1135}]},{rabbit_queue_index,segment_entries_foldr,3,[{file,"src/rabbit_queue_index.erl"},{line,1091}]},{lists,foldr,3,[{file,"lists.erl"},{line,1276}]},{rabbit_queue_index,scan_queue_segments,3,[{file,"src/rabbit_queue_index.erl"},{line,725}]}]}]
    neighbour: [{pid,<0.336.0>},{registered_name,[]},{initial_call,{gatherer,init,['Argument__1']}},{current_function,{gen_server2,process_next_msg,1}},{ancestors,[<0.332.0>,<0.324.0>,<0.323.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.262.0>]},{message_queue_len,2},{links,[<0.280.0>,<0.332.0>,<0.281.0>,<0.278.0>,<0.279.0>]},{trap_exit,false},{status,runnable},{heap_size,987},{stack_size,8},{reductions,73223},{current_stacktrace,[{gen_server2,process_next_msg,1,[{file,"src/gen_server2.erl"},{line,666}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}]
2018-09-11 09:39:42 =CRASH REPORT====
  crasher:
    initial call: rabbit_msg_store:init/1
    pid: <0.332.0>
    registered_name: []
    exception exit: {{{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]},{gen_server2,call,[<0.336.0>,out,infinity]}},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,589}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
    ancestors: [<0.324.0>,<0.323.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.262.0>]
    message_queue_len: 1
    messages: [{'EXIT',<0.336.0>,{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]}}]
    links: [<0.335.0>,<0.324.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 57377
  neighbours:
    neighbour: [{pid,<0.335.0>},{registered_name,[]},{initial_call,{rabbit_msg_store_gc,init,['Argument__1']}},{current_function,{gen_server2,process_next_msg,1}},{ancestors,[<0.332.0>,<0.324.0>,<0.323.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.262.0>]},{message_queue_len,0},{links,[<0.332.0>]},{trap_exit,false},{status,waiting},{heap_size,987},{stack_size,8},{reductions,174},{current_stacktrace,[{gen_server2,process_next_msg,1,[{file,"src/gen_server2.erl"},{line,666}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}]
2018-09-11 09:39:42 =SUPERVISOR REPORT====
     Supervisor: {local,worker_pool_sup}
     Context:    child_terminated
     Reason:     {function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]}
     Offender:   [{pid,<0.281.0>},{id,4},{mfargs,{worker_pool_worker,start_link,[worker_pool]}},{restart_type,transient},{shutdown,4294967295},{child_type,worker}]

2018-09-11 09:39:42 =CRASH REPORT====
  crasher:
initial call: rabbit_vhost_process:init/1
pid: <0.325.0>
registered_name: []
exception exit: {{error,{{{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]},{gen_server2,call,[<0.336.0>,out,infinity]}},{child,undefined,msg_store_persistent,{rabbit_msg_store,start_link,[msg_store_persistent,"c:/Users/dfpsb/AppData/Roaming/RabbitMQ/db/RABBIT~1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",[],{#Fun<rabbit_queue_index.2.122888644>,{start,[{resource,<<"/">>,queue,<<"DF-9ID59RK-WS.InterchangeFragtbrevEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Test1.InterchangeFragtbrevEnvelope.RET">>},{resource,<<"/">>,queue,<<"Test2.DfLoggingEvent.Debug">>},{resource,<<"/">>,queue,<<"DF-9ID59RK-WS.InterchangeTurEnvelope.DFMobil">>},{resource,<<"/">>,queue,<<"Paw.DfLoggingEvent.Debug">>},{resource,<<"/">>,queue,<<"DevUnitTest.TruckLoadingEnvelope.UnitTest">>},{resource,<<"/">>,queue,<<"Test1.InterchangeFragtbrevEnvelope.RET_error">>},{resource,<<"/">>,queue,<<"Paw.InterchangeFragtbrevEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"DevUnitTest.TestMsg.UnitTest_error">>},{resource,<<"/">>,queue,<<"DevUnitTest.TestMsg.UnitTest">>},{resource,<<"/">>,queue,<<"Paw.InterchangeTurEnvelope.DFMobil">>},{resource,<<"/">>,queue,<<"Test2.InterchangeFragtbrevEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Paw.InterchangeFragtbrevEnvelope.TurPlan_error">>},{resource,<<"/">>,queue,<<"Paw.TruckLoadingEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Test2.InterchangeFragtbrevEnvelope.TurPlan_error">>},{resource,<<"/">>,queue,<<"Test2.TruckLoadingEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Paw.DfLoggingEvent.Warning">>},{resource,<<"/">>,queue,<<"DF-9ID59RK-WS.InterchangeFragtbrevEnvelope.RET">>},{resource,<<"/">>,queue,<<"Test2.InterchangeTurEnvelope.DFMobil">>},{resource,<<"/">>,queue,<<"Test2.DfLoggingEvent.Warning">>}]}}]},transient,30000,worker,[rabbit_msg_store]}}},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,581}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
ancestors: [<0.323.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.262.0>]
message_queue_len: 0
messages: []
links: [<0.323.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 10958
stack_size: 27
reductions: 63314
  neighbours:
2018-09-11 09:39:42 =SUPERVISOR REPORT====
 Supervisor: {<0.323.0>,rabbit_vhost_sup_wrapper}
 Context:    start_error
 Reason:     {error,{{{function_clause,[{rabbit_queue_index,journal_minus_segment1,[{{true,<<172,190,166,92,192,205,125,125,36,223,114,188,53,139,128,108,0,0,0,0,0,0,0,0,0,0,26,151>>,<<>>},no_del,no_ack},{{true,<<89,173,78,227,188,37,119,171,231,189,220,236,244,79,138,177,0,0,0,0,0,0,0,0,0,0,23,40>>,<<>>},no_del,no_ack}],[{file,"src/rabbit_queue_index.erl"},{line,1231}]},{rabbit_queue_index,'-journal_minus_segment/3-fun-0-',4,[{file,"src/rabbit_queue_index.erl"},{line,1208}]},{array,sparse_foldl_3,7,[{file,"array.erl"},{line,1684}]},{array,sparse_foldl_2,9,[{file,"array.erl"},{line,1678}]},{rabbit_queue_index,'-recover_journal/1-fun-0-',1,[{file,"src/rabbit_queue_index.erl"},{line,915}]},{lists,map,2,[{file,"lists.erl"},{line,1239}]},{rabbit_queue_index,segment_map,2,[{file,"src/rabbit_queue_index.erl"},{line,1039}]},{rabbit_queue_index,recover_journal,1,[{file,"src/rabbit_queue_index.erl"},{line,906}]}]},{gen_server2,call,[<0.336.0>,out,infinity]}},{child,undefined,msg_store_persistent,{rabbit_msg_store,start_link,[msg_store_persistent,"c:/Users/dfpsb/AppData/Roaming/RabbitMQ/db/RABBIT~1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",[],{#Fun<rabbit_queue_index.2.122888644>,{start,[{resource,<<"/">>,queue,<<"DF-9ID59RK-WS.InterchangeFragtbrevEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Test1.InterchangeFragtbrevEnvelope.RET">>},{resource,<<"/">>,queue,<<"Test2.DfLoggingEvent.Debug">>},{resource,<<"/">>,queue,<<"DF-9ID59RK-WS.InterchangeTurEnvelope.DFMobil">>},{resource,<<"/">>,queue,<<"Paw.DfLoggingEvent.Debug">>},{resource,<<"/">>,queue,<<"DevUnitTest.TruckLoadingEnvelope.UnitTest">>},{resource,<<"/">>,queue,<<"Test1.InterchangeFragtbrevEnvelope.RET_error">>},{resource,<<"/">>,queue,<<"Paw.InterchangeFragtbrevEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"DevUnitTest.TestMsg.UnitTest_error">>},{resource,<<"/">>,queue,<<"DevUnitTest.TestMsg.UnitTest">>},{resource,<<"/">>,queue,<<"Paw.InterchangeTurEnvelope.DFMobil">>},{resource,<<"/">>,queue,<<"Test2.InterchangeFragtbrevEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Paw.InterchangeFragtbrevEnvelope.TurPlan_error">>},{resource,<<"/">>,queue,<<"Paw.TruckLoadingEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Test2.InterchangeFragtbrevEnvelope.TurPlan_error">>},{resource,<<"/">>,queue,<<"Test2.TruckLoadingEnvelope.TurPlan">>},{resource,<<"/">>,queue,<<"Paw.DfLoggingEvent.Warning">>},{resource,<<"/">>,queue,<<"DF-9ID59RK-WS.InterchangeFragtbrevEnvelope.RET">>},{resource,<<"/">>,queue,<<"Test2.InterchangeTurEnvelope.DFMobil">>},{resource,<<"/">>,queue,<<"Test2.DfLoggingEvent.Warning">>}]}}]},transient,30000,worker,[rabbit_msg_store]}}}
 Offender:   [{pid,undefined},{id,rabbit_vhost_process},{mfargs,{rabbit_vhost_process,start_link,[<<"/">>]}},{restart_type,permanent},{shutdown,30000},{child_type,worker}]

Answer 1

I figured out what was going on. Someone on my team (me) was creating an unprecedented amount of connections to RabbitMQ. For each connection a file handler is created to one or more files, not sure which. The OS (Windows in my case) has a file handler limit, not sure what the limit is, but when reached, an error is thrown. This corrupted the virtual host (/) and I had to delete it and create it again. Good thing this wasn't production, because then all the messages would've been gone.

Edit (June 2020):

There might be a fix for this so it can't happen again. A virtual host can be limited in how many connections they allow. This way Windows won't faulter on the many file handles. Look in the management portal -> Admin -> Limits, set max-connections to a number you think it plausible for your setup.