How can I limit memory acquired with `malloc()` without also limiting stack?

Question

I'm trying to keep student code from running wild with allocations and dragging my test machine to a halt. I've tried

setrlimit(RLIMIT_DATA, r);

where r is a struct holding the limits. But unfortunately although this limit stops brk and sbrk from allocating, the C library just fails over to mmap and keeps right on allocating.

I've also tried

setrlimit(RLIMIT_AS, r)

and this stops the process in its tracks, but this remedy is too severe—it is impossible for the process to recover from the ENOMEM error because there's no stack space for the calls the code makes on encountering a NULL value returned from malloc().

I have limited controls over the binaries, so if it's possible to do with a system call, I'd prefer that. But I need some means of capping allocation without destroying the process's ability to recover. Does anyone have suggestions?

UPDATE: I found something called failmalloc, but it is not very sophisticated, and although I can cause a failure with it, I always get a segfault that gdb cannot diagnose.

FURTHER UPDATE: I found that setrlimit(RLIMIT_AS, r) does seem to do the job I want, at least in some cases—the segfaults that were occurring afterward were caused by a fault in an unrelated module. Unless somebody comes up with something interesting (or a reason to keep the question), I will probably delete the question.

Answer 1

Building on the idea used by failmalloc, you could use the LD_PRELOAD^* environment variable and function interposition to build a wrapper around malloc() and impose any limitations there.

You would need to dynamically load a pointer to the original malloc() using dlsym(). You cannot directly call the original malloc() from the wrapper because it will be interpreted as a recursive call to the wrapper itself.

#define _GNU_SOURCE
#include <stdio.h>
#include <stdint.h>
#include <dlfcn.h>

void * malloc(size_t size)
{
   static void * (*func)(size_t) = NULL;
   void * ret;

   if (!func)
   {
      /* get reference to original (libc provided) malloc */
      func = (void *(*)(size_t)) dlsym(RTLD_NEXT, "malloc");
   }

   /* impose any necessary restrictions before calling malloc */
   ...

   /* call original malloc */
   ret = func(size);

   /* impose any necessary restrictions after calling malloc */
   ...

   return ret;
}

* Note that LD_PRELOAD must specify the full path to the interposer library, and that library interposition is disabled for setuid programs in order to prevent security problems.

---

An alternative to using dlsym() would be to use the GNU linker --wrap symbol option.