Erlang "system" memory section keeps growing

Question

I have an application with the following pattern:

• 2 long running processes that go into hibernate after some idle time and their memory consumption goes down as expected
• N (0 < N < 100) worker processes that do some work and hibernate when idle more than 10 seconds or terminate if idle more than two hours
• during the night, when there is no activity the process memory goes back to almost the same value that was at the application start, which is expected as all the workers have died.

The issue is that "system" section keeps growing (around 1GB/week).

My question is how can I debug what is stored there or who's allocating memory in that area and is not freeing it.

I've already tested lists:keysearch/3 and it doesn't seem to leak memory, as that is the only native thing I'm using (no ports, no drivers, no NIFs, no BIFs, nothing). Erlang version is R15B03.

Here is the current erlang:memory() output (slight traffic, app started on Feb 03):

[{total,378865650},
{processes,100727351},
{processes_used,100489511},
{system,278138299},
{atom,1123505},
{atom_used,1106100},
{binary,4493504},
{code,7960564},
{ets,489944},
{maximum,402598426}]

This is a 64-bit system. As you can see, "system" section has ~270MB and "processes" is at around 100MB (that drops down to ~16MB during the night).

Answer 1

It seems that I've found the issue.

I have a "process_killer" gen_server where processes can subscribe for periodic GC or kill. Its subscribe functions are called on each message received by some processes to postpone the GC/kill (something like re-arm).

This process performs an erlang:monitor if not already monitored to catch a dead process and remove it from watch list. If I comment our the re-subscription line on each handled message, "system" area seems to behave normally. That means it is a bug in my process_killer that does leak monitor refs (remember you can call erlang:monitor multiple times and each call creates a reference).

I was lead to this idea because I've tested a simple module which was calling erlang:monitor in a loop and I have seen ~13 bytes "system" area grow on each call.

The workers themselves were OK because they would die anyway taking their monitors along with them. There is one long running (starts with the app, stops with the app) process that dispatches all the messages to the workers that was calling GC re-arm on each received message, so we're talking about tens of thousands of monitors spawned per hour and never released.

I'm writing this answer here for future reference.

TL;DR; make sure you are not leaking monitor refs on a long running process.