How can I force MacOS to release MADV_FREE'd pages?

Question

My program has a custom allocator which gets memory from the OS using mmap(MAP_ANON | MAP_PRIVATE). When it no longer needs memory, the allocator calls either munmap or madvise(MADV_FREE). MADV_FREE keeps the mapping around, but tells the OS that it can throw away the physical pages associated with the mapping.

Calling MADV_FREE on pages you're going to need again eventually is much faster than calling munmap and later calling mmap again.

This almost works perfectly for me. The only problem is that, on MacOS, MADV_FREE is very lazy about getting rid of the pages I've asked it to free. In fact, it only gets rid of them when there's memory pressure from another application. Until it gets rid of the pages I've freed, MacOS reports that my program is still using that memory; in the Activity Monitor, its "Real Memory" column doesn't reflect the freed memory.

This makes it difficult for me to measure how much memory my program is actually using. (This difficulty in measuring RSS is keeping us from landing the custom allocator on 10.5.)

I could allocate a whole bunch of memory to force the OS to free up these pages, but in addition to taking a long time, that could have other side-effects, such as causing parts of my program to be paged out to disk.

On a lark, I tried the purge command, but that has no effect.

How can I force MacOS to clean out these MADV_FREE'd pages? Or, how can I ask MacOS how many MADV_FREE'd pages my process has in memory?

Here's a test program, if it helps. The Activity Monitor's "Real Memory" column shows 512MB after the program goes to sleep. On my Linux box, top shows 256MB of RSS, as desired.

#include <sys/mman.h>
#include <stdio.h>
#include <unistd.h>

#define SIZE (512 * 1024 * 1024)

// We use MADV_FREE on Mac and MADV_DONTNEED on Linux.
#ifndef MADV_FREE
#define MADV_FREE MADV_DONTNEED
#endif

int main()
{
  char *x = mmap(0, SIZE, PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE, -1, 0);

  // Touch each page we mmap'ed so it gets a physical page.
  int i;
  for (i = 0; i < SIZE; i += 1024) {
    x[i] = i;
  }

  madvise(x, SIZE / 2, MADV_FREE);

  fprintf(stderr, "Sleeping.  Now check my RSS.  Hopefully it's %dMB.\n", SIZE / (2 * 1024 * 1024));
  sleep(1024);
  return 0;
}

Answer 1

mprotect(addr, length, PROT_NONE);
mprotect(addr, length, PROT_READ | PROT_WRITE);

Note as you say, madvise is lazier, and that is probably better for performance (just in case anyone is tempted to use this for performance rather than measurement).

Answer 2

Use MADV_FREE_REUSABLE on macOS. According to Apple's magazine_malloc implementation:

On OS X we use MADV_FREE_REUSABLE, which signals the kernel to remove the given pages from the memory statistics for our process. However, on returning that memory to use we have to signal that it has been reused.

https://opensource.apple.com/source/libmalloc/libmalloc-53.1.1/src/magazine_malloc.c.auto.html

Chromium, for example, also uses it:

MADV_FREE_REUSABLE is similar to MADV_FREE, but also marks the pages with the reusable bit, which allows both Activity Monitor and memory-infra to correctly track the pages.

https://github.com/chromium/chromium/blob/master/base/memory/discardable_shared_memory.cc#L377