Let me put my question as simple as below. Mine is a network router software built in erlang, but at a particular scenario I am observing very high memory growth as shown by VM.
I have one process which receives binary packet from some other process from socket.
This process, parses the binary packet and passes the binary packet to a gen_server (handle_cast is called)
The gen_server again stores some information in ETS table and send the packet to the peer server.
When the peer server responds back the entry from the ETS is deleted and the gen_server responds back to the first process
Also if the first process (which sent packet to gen_server) gets timedout after 5 seconds waiting for response from gen_server , it also deletes the ETS entry in the gen_server and exits.Now I am observing high memory growth when lots of events gets timed out (due to unavailability of peer server) and from what i have researched its the "**binary**" and "**processes_used**" given by erlang:memory command thats using most of the memory.
but the same is not true when events are processed successfully.
The memory lost can be basically only in three places:
The state of your gen_server
Your processes mailboxes
see to it that there is some way to always drain unmatched messages (for gen_server handle_info
callback) in normal receive
s a Any ->
clause.
if the mailbox only fills up temporarily its probably because of the receiving process being too slow for the rate of messages produced. This is usually a problem for asynchronous communication. If its only temporary bursts that don't break anything this could be intended.
In this case you can either optimize the receiving process
or fix your protocol to use fewer messages
if you have multiple functions that receive some messages, make sure all receiving parts are being called regularly. Dont forget the Any ->
clauses.
Be aware that while you are processing in a gen_servers callback no messages will be received, so if you need more time in a callback that would be necessary asyncronous messages might pile up (e.g. random message arrival + fixed processing time builds a unbounded growing queue, for details see Queueing theory
In your ETS table
trigger GC by hand and see what happens with memory.
[garbage_collect(Pid) || Pid <- processes()]
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With