How to minimize memory consumption of c++ program using copy-on-write?

Question

I'm working on an application which forks itself up to 8 times for parallelism. Each fork has a full copy of the memory space from the original process at the time of the fork. The forks are quick because Linux shares the pages between the processes and only creates new ones when they are modified. The growth in memory consumption, in practice, seems to be about 3x for my application. Any suggestions for tools or techniques to use in identifying changes that will reduce that growth?

One thought is to look at the page fragmentation of the modified pages. There's also just the brute force examination of what's allocated in the forked processes. In either case, what techniques or tools can you recommend for performing that analysis?

Keep in mind that the program takes several hours to complete even with parallelism and has a memory footprint of up to 1TB so instrumentation options are limited.

Answer 1

You can use vmstat, systemtap and or glibc's malloc-hooks to monitor consumption. You can use perf to see where faults are occurring in order to understand the actual impact of the consumption.

If your application faces TLB pressure when you use large memory pools (e.g. you are streaming through a very large volume of data), you can use huge/large pages to mitigate the overhead of 4k pages.

You can also use madvise to tell the kernel from within your processes what you're probably going to do with the memory that was allocated.