Solving large binaries leak

Question

I have an elixir/OTP application that crashes in production due to an out-of-memory issue. The function that cause the crash is called every 6 hours, in a dedicated process. It takes several minutes (~30) to run and looks like this:

def entry_point do
  get_jobs_to_scrape()
  |> Task.async_stream(&scrape/1)
  |> Stream.map(&persist/1)
  |> Stream.run()
end

On my local machine I see a constant growth in large binaries memory consumption when the function runs:

Note that when I manually trigger garbage collection on the process that runs the function the memory consumption drops significantly, so it's definitely not a problem with several different processes unable to GC, but only one that doesn't GC properly. In addition, it's important to say that every few minutes the process does manage to GC, but sometimes it's not enough. The production server has only 1GB RAM and it crashes before the GC kicks in.

Trying to solve the issue I came across Erlang in Anger (see pages 66-67). One suggestion is to put all of large binaries manipulations in one-off processes. The return value of the scrape function is a map that contains the large binaries. Therefore, they are shared between the Task.async_stream "workers" and the process that runs the function. So, in theory, I could put the persist together with the scrape inside the Task.async_stream. I prefer not to do so, and keep the calls to persist synchronized through the process.

Another suggestion is to call :erlang.garbage_collect periodically. It looks like it solves the problem but feels way too hacky. The author also doesn't recommend that. Here's my current solution:

def entry_point do
  my_pid = self()
  Task.async(fn -> periodically_gc(my_pid) end)
  # The rest of the function as before...
end

defp periodically_gc(pid) do
  Process.sleep(30_000)
  if Process.alive?(pid) do
    :erlang.garbage_collect(pid)
    periodically_gc(pid)
  end
end

And the resulted memory load:

I don't quite understand how the other suggestions in the book fit the problem.

What would you recommend in that case? Keep the hacky solution or there are better options.

Answer 1

The erlang virtual machine has a garbage collection mechanism that, by default, is optimised for short lived data. A short lived process may not be garbage collected at all until it dies, and most garbage collection runs only checks newly added items. Items that has survived a GC run will not be checked again until a full sweep is done.

I would suggest that you try adjusting the fullsweep_after flag. It can be set globally through :erlang.system_flag(:fullsweep_after, value) or for your specific process using :erlang.spawn_opt/4.

From the docs:

The Erlang runtime system uses a generational garbage collection scheme, using an "old heap" for data that has survived at least one garbage collection. When there is no more room on the old heap, a fullsweep garbage collection is done.

Option fullsweep_after makes it possible to specify the maximum number of generational collections before forcing a fullsweep, even if there is room on the old heap. Setting the number to zero disables the general collection algorithm, that is, all live data is copied at every garbage collection.

A few cases when it can be useful to change fullsweep_after:

• If binaries that are no longer used are to be thrown away as soon as possible. (Set Number to zero.)
• A process that mostly have short-lived data is fullsweeped seldom or never, that is, the old heap contains mostly garbage. To ensure a fullsweep occasionally, set Number to a suitable value, such as 10 or 20.
• In embedded systems with a limited amount of RAM and no virtual memory, you might want to preserve memory by setting Number to zero. (The value can be set globally, see erlang:system_flag/2.)

The default value is 65535 (unless you already changed it through the environment variable ERL_FULLSWEEP_AFTER), so any lower value will make the garbage collection more aggressive.

This is a good read on the subject: https://www.erlang-solutions.com/blog/erlang-19-0-garbage-collector.html