Processes resources not limited by setrlimit

Question

I wrote a simple program to restrict it's data size to 65Kb and to verify the same i am allocating a dummy memory of more than 65Kb and logically if i am doing all correct (as below) malloc call should fail, isn't it?

#include <sys/resource.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

int main (int argc, char *argv[])
{
  struct rlimit limit;


  /* Get max data size . */
  if (getrlimit(RLIMIT_DATA, &limit) != 0) {
    printf("getrlimit() failed with errno=%d\n", errno);
    return 1;
  }

  printf("The soft limit is %lu\n", limit.rlim_cur);
  printf("The hard limit is %lu\n", limit.rlim_max);

  limit.rlim_cur = 65 * 1024;
  limit.rlim_max = 65 * 1024;

  if (setrlimit(RLIMIT_DATA, &limit) != 0) {
    printf("setrlimit() failed with errno=%d\n", errno);
    return 1;
  }

  if (getrlimit(RLIMIT_DATA, &limit) != 0) {
    printf("getrlimit() failed with errno=%d\n", errno);
    return 1;
  }

  printf("The soft limit is %lu\n", limit.rlim_cur);
  printf("The hard limit is %lu\n", limit.rlim_max);
  system("bash -c 'ulimit -a'");
    int *new2 = NULL;
    new2 = malloc(66666666);
    if (new2 == NULL)
    {
        printf("malloc failed\n");
        return;
    }
    else
    {
        printf("success\n");
    }

  return 0;
}

Surprisingly, the ouput is something like this -

The soft limit is 4294967295
The hard limit is 4294967295
The soft limit is 66560
The hard limit is 66560
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) 65
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 14895
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 14895
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
success

Am i doing wrong in any way? Please drop your inputs. Thanks!

Answer 1

From the setrlimit man page:

RLIMIT_DATA

The maximum size of the process's data segment (initialized data, uninitialized data, and heap). This limit affects calls to brk(2) and sbrk(2), which fail with the error ENOMEM upon encountering the soft limit of this resource.

Specifically, that resource does not apply to memory obtained via mmap. Internally malloc uses various mechanisms for obtaining new memory. In this case you will find that it used mmap and not sbrk or brk. You can verify this by dumping the system calls from your program with strace.

To achieve what you want use the RLIMIT_AS resource instead.