Make -j RAM limits

Question

Is there any way to force make -j to not over consume my RAM? I work on a dev team, and we have different hardware sets, so -j8 may not be optimal for everyone. However, make -j uses too much RAM for me, and spills over into swap, which can take down my entire system. How can I avoid this?

Ideally, I would want make to watch the system load and stop spawning new threads, wait for some to complete, and continue on.

Answer 1

The somewhat simple solution would be for each workstation to have an environment variable that is suited to what that hardware can handle. Have the makefile read this environment variable and pass it to the -j option. How to get gnu make to read env variables.

Also, if the build process has many steps and takes a long time have make re-read the environment variable so that during the build you can reduce / increase resource usage.

Also, maybe have a service/application running on the workstation that does the monitoring of resource usage and modify the environment variable instead of trying to have make do it...

Answer 2

Is it possible that there is some confusion what make -j does? (at least I had it wrong for a long time...). I assumed that -j without options will adapt to the number of CPUs, but it doesn't - it simply doesn't apply any limit. This means that for big projects it will create a huge number of processes (just run "top" to see...), and possibly use up all the RAM. I stumbled on this when the "make -j" of my project used all of my 16Gb of RAM and failed, while "make -j 8" topped out at 2.5 Gb RAM usage (on 8 cores, load is close to 100% in both cases).

In terms of efficiency, I think using a limit equal to or bigger than the maximum number of CPUs you expect is more efficient that no limit, as the scheduling of thousands of processes has some overhead. Total number of processes created should be constant, but as "-j" creates a lot of them at once, memory allocation might become a problem. Even setting the limit to twice the number of CPUs should still be more conservative that not setting a limit at all.

PS: After some more digging I came up with a neat solution - just read out the number of processors and use that as the -j option:

make -j `nproc`