Limit available memory to a program

Question

I am doing performance analysis on a multi-threaded program that uses all the memory that is available in the system. My OS is Ubuntu 18.04. I'm trying to limit the available memory to e.g 32GB even though my server may have 128GB of memory available. Haven't been able to find a reliable solution. Seems like ulimit is not exactly doing what I'm looking for. I can also clog up memory by another process (e.g a controllable process that will consume 64GB of RAM). But even for that purpose I'm not sure how to reliably clog up the memory.

Would appreciate your thoughts.

Answer 1

cgroups is the feature or modern linux kernels which allow you to limit resources like memory for group of processes (or for single process with threads). More about cgroups: https://en.wikipedia.org/wiki/Cgroups https://man7.org/linux/man-pages/man7/cgroups.7.html

The cgroups feature should be already enabled in your ubuntu 18.04 kernels. There are some descriptions how to use cgroups to limit memory:

• for systemd managed service: https://www.paranoids.at/cgroup-ubuntu-18-04-howto/,
• for docker containers docker run --memory=1G ….
• for custom process tree with cgcreate/cgset/cgexec commands: https://dev.to/vga/how-to-see-and-limit-memory-consumption-of-an-application-5bfl something like

# Create a group for memory named “limited_group_1”
cgcreate -g "memory:limited_group_1" -t USERNAME:GROUPNAME

# Specify memory limit to 1G for this group
cgset -r memory.limit_in_bytes=1G "limited_group_1"

# Launch the application in this group
cgexec -g "memory:limited_group_1" ./YOUR_APPLICATION

# If needed, we can remove the group
cgdelete "memory:limited_group_1"

https://unix.stackexchange.com/questions/44985/limit-memory-usage-for-a-single-linux-process/279175#279175 was also mentioned in https://dev.to/vga/how-to-see-and-limit-memory-consumption-of-an-application-5bfl

PS: Default memory allocators in older glibc versions (malloc, new) has awful behavior for freed regions: they are not returned back without periodic malloc_trim() library calls. You should try to link your application with libjemalloc or libtcmalloc which will replace malloc implementation of glibc with some code better in memory returning.