How to set memory use limit when writing C program and what happens if once it exceeds this limit?

Question

I am writing a C program on linux and I wonder:

1. How to limit the total memory my c program consumes?

2. If I set a memory limit to my c program, say 32M, what happens if it requires much more memory than 32M?

Answer 1

You should use the setrlimit system call, with the RLIMIT_DATA and RLIMIT_STACK resources to limit the heap and stack sizes respectively. It is tempting to use RLIMIT_AS or RLIMIT_RSS but you will discover that they do not work reliably on many old Linux kernels, and I see no sign on the kernel mailing list that the problems have been resolved in the latest kernels. One problem relates to how mmap'd memory is counted or rather not counted toward the limit totals. Since glibc malloc uses mmap for large allocations, even programs that don't call mmap directly can exceed the limits.

If you exceed the RLIMIT_STACK limit (call stack too deep, or allocate too many variables on the stack), your program will receive a SIGSEGV. If you try to extend the data segment past the RLIMIT_DATA limit (brk, sbrk or some intermediary like malloc), the attempt will fail. brk or sbrk will return < 0 and malloc will return a null pointer.