How Limit memory usage for a single Linux process and not kill the process

Question

How Limit memory usage for a single Linux process and not kill the process.

I know ulimit can limit memory usage, but if exceed the limit, will kill the process.

Is there any other command or shell can limit memory usage and not kill the process?

Answer 1

Another way besides setrlimit, which can be set using the ulimit utility:

$ ulimit -Sv 500000 # Set ~500 mb limit

is to use Linux's control groups, because it limits a process's (or group of processes') allocation of physical memory distinctly from virtual memory. For example:

$ cgcreate -g memory:/myGroup
$ echo $(( 500 * 1024 * 1024 )) > /sys/fs/cgroup/memory/myGroup/memory.limit_in_bytes
$ echo $(( 5000 * 1024 * 1024 )) > /sys/fs/cgroup/memory/myGroupmemory.memsw.limit_in_bytes

will create a control group named "myGroup", cap the set of processes run under myGroup up to 500 MB of physical memory and up to 5000 MB of swap. To run a process under the control group:

$ cgexec -g memory:myGroup COMMAND

Note: For what I can understand setrlimit will limit the virtual memory, although with cgroups you can limit the physical memory.

Answer 2

I believe you are wrong on thinking that a limit set with setrlimit(2) will always kill the process.

Indeed, if the stack space is exceeded (RLIMIT_STACK), the process would be killed (with SIGSEGV).

But if it is heap memory (RLIMIT_DATA or RLIMIT_AS), mmap(2) would fail. If it has been called from malloc(3) or friends, that malloc would fail.

^{Some Linux systems are configured with memory overcommit.
This is a sysadmin issue: echo 0 > /proc/sys/vm/overcommit_memory}

The moral of the story is that you should always check result of malloc, at least like

struct mystruct_st *ptr = malloc(sizeof(struct mystruct_st));
if (!ptr) { perror("mystruct allocation"); exit(EXIT_FAILURE); }

^{Of course, sophisticated applications could handle "out-of-memory" conditions more wisely, but it is difficult to get right.}

Some people incorrectly assumes that malloc does not fail (this is wrong). But that is their mistake. Then they dereference a NULL pointer, getting a SIGSEGV that they deserve.

You could consider catching SIGSEGV with some processor-specific code, see this answer. Unless you are a guru, don't do that.