When does the MacOSX 'free' library call madvise, and is there any way to control it?

Question

I've got a C++ program that is notably slower on OSX 10.8.2 than on Linux. Profiling shows that the reason is that calls to free (that result from STL operations, FWIW), are much slower on OSX, because they go and call madvise, and real time gets consumed in there.

Is there any way to modulate this behavior of OS/X?

Answer 1

Well, yes!

I had horrible performance issues with malloc/free in Linux and started looking for a replacement. Two options came to mind tbbmalloc (part of Intel TBB which is free BTW) and Google malloc. After extensive testing it wasn't clear which was faster (of the two) but both were significantly faster than LIBC's implementation.

I went with tbbmalloc since it was working smoother, google malloc had a bug that caused virtual memory to be very large (reserved but not committed) which was very bad for my app (IT daemons would kill it).

The good:

• Much better performance than libc's malloc. Was 3x-300x in STL heavy app.
• Simple integration. No code change. Add/change 1 line the executable's makefile. No change to SOs.

The bad:

• Mem checkers will not with replacements. for memchk/valgrind/etc. revert to the original malloc.
• App would take 10-30% more memory.

The app I developed was a CAD application that used 10s of GBs, building and destroying 10s of millions various structures (lots of STL maps, vectors, hash_maps).

How to do this:

In the linker command, add -ltbbmalloc and make sure the library is in the lib search path (-L flag).