How to make OS X malloc automatically abort on failure?

Question

The Matasano blog calls “Checking the return value of malloc()” a “C Programming Anti-Idiom.” Instead malloc() should automatically call abort() for you if it fails. The argument is that, since you’ll usually want to abort the program if malloc() fails, that should be the default behaviour instead of something you have to laboriously type—or maybe forget to type!—every time.

Without getting into the merits of the idea, what’s the easiest way to set this up? I’m looking for something that would automatically detect memory allocation failures by other library functions such as asprintf() too. A portable solution would be wonderful, but I’d also be happy with something mac-specific.

---

Summarizing the best answers below:

Mac run-time solution

Set the MallocErrorAbort=1 environment variable before running your program. Automatically works for all memory allocation functions.

Mac/linux run-time solution

Use a dynamic library shim to load a custom malloc() wrapper at runtime with LD_PRELOAD or DYLD_INSERT_LIBRARIES. You will likely want to wrap calloc(), realloc(), &c. as well.

Mac/linux compiled solution

Define your own malloc() and free() functions, and access the system versions using dyld(RTLD_NEXT, "malloc") as shown here. Again, you will likely want to wrap calloc(), realloc(), &c. as well.

#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

void *(*system_malloc)(size_t) = NULL;

void* malloc(size_t bytes) {
    if (system_malloc == NULL) {
        system_malloc = dlsym(RTLD_NEXT, "malloc");
    }
    void* ret = system_malloc(bytes);
    if (ret == NULL) {
        perror("malloc failed, aborting");
        abort();
    }
    return ret;
}

int main() {
    void* m = malloc(10000000000000000l);
    if (m == NULL) {
        perror("malloc failed, program still running");
    }
    return 0;
}

Linux compiled solution

Use __malloc_hook and __realloc_hook as described in the glibc manual.

Mac compiled solution

Use the malloc_default_zone() function to access the heap’s data structure, unprotect the memory page, and install a hook in zone->malloc:

#include <malloc/malloc.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <unistd.h>

static void* (*system_malloc)(struct _malloc_zone_t *zone, size_t size);
static void* my_malloc(struct _malloc_zone_t *zone, size_t size) {
    void* ret = system_malloc(zone, size);
    if (ret == NULL) {
        perror("malloc failed, aborting");
        abort();
    }
    return ret;
}

int main() {
    malloc_zone_t *zone = malloc_default_zone();
    if (zone->version != 8) {
        fprintf(stderr, "Unknown malloc zone version %d\n", zone->version);
        abort();
    }
    system_malloc = zone->malloc;
    if (mprotect(zone, getpagesize(), PROT_READ | PROT_WRITE) != 0) {
        perror("munprotect failed");
        abort();
    }
    zone->malloc = my_malloc;
    if (mprotect(zone, getpagesize(), PROT_READ) != 0) {
        perror("mprotect failed");
        abort();
    }

    void* m = malloc(10000000000000000l);
    if (m == NULL) {
        perror("malloc failed, program still running");
    }
    return 0;
}

For completeness you will likely want to wrap calloc(), realloc(), and the other functions defined in malloc_zone_t in /usr/include/malloc/malloc.h as well.

Answer 1

Just wrap malloc() in some my_malloc() function that does this instead. In a lot of cases it's actually possible to handle not being able to allocate memory so this type of behaviour would be undesirable. It's easy to add functionality to malloc() but not to remove it, which is probably why it behaves this way.

Another thing to keep in mind is that this is a library you're calling into. Would you like to make a library call and have the library kill your application without you being able to have a say in it?

I guess I missed the part about asprintf but libc exports some hooks you can use (what valgrind does essentially) that let you override the malloc behavior. Here's a reference to the hooks themselves, if you know C well enough it's not hard to do.

http://www.gnu.org/savannah-checkouts/gnu/libc/manual/html_node/Hooks-for-Malloc.html

Answer 2

man malloc on my Mac gives the following information. It looks like you want MallocErrorAbort.

ENVIRONMENT

The following environment variables change the behavior of the allocation-related functions.

• MallocLogFile <f>
  Create/append messages to the given file path <f> instead of writing to the standard error.

• MallocGuardEdges
  If set, add a guard page before and after each large block.

• MallocDoNotProtectPrelude
  If set, do not add a guard page before large blocks, even if the MallocGuardEdges environment variable is set.

• MallocDoNotProtectPostlude
  If set, do not add a guard page after large blocks, even if the MallocGuardEdges environment variable is set.

• MallocStackLogging
  If set, record all stacks, so that tools like leaks can be used.

• MallocStackLoggingNoCompact
  If set, record all stacks in a manner that is compatible with the malloc_history program.

• MallocStackLoggingDirectory
  If set, records stack logs to the directory specified instead of saving them to the default location (/tmp).

• MallocScribble
  If set, fill memory that has been allocated with 0xaa bytes. This increases the likelihood that a program making assumptions about the contents of freshly allocated memory will fail. Also if set, fill memory that has been deallocated with 0x55 bytes. This increases the likelihood that a program will fail due to accessing memory that is no longer allocated.

• MallocCheckHeapStart <s>
  If set, specifies the number of allocations <s> to wait before beginning periodic heap checks every <n> as specified by MallocCheckHeapEach. If MallocCheckHeapStart is set but MallocCheckHeapEach is not specified, the default check repetition is 1000.

• MallocCheckHeapEach <n>
  If set, run a consistency check on the heap every <n> operations. MallocCheckHeapEach is only meaningful if MallocCheckHeapStart is also set.

• MallocCheckHeapSleep <t>
  Sets the number of seconds to sleep (waiting for a debugger to attach) when MallocCheckHeapStart is set and a heap corruption is detected. The default is 100 seconds. Setting this to zero means not to sleep at all. Setting this to a negative number means to sleep (for the positive number of seconds) only the very first time a heap corruption is detected.

• MallocCheckHeapAbort <b>
  When MallocCheckHeapStart is set and this is set to a non-zero value, causes abort(3) to be called if a heap corruption is detected, instead of any sleeping.

• MallocErrorAbort
  If set, causes abort(3) to be called if an error was encountered in malloc(3) or free(3), such as a calling free(3) on a pointer previously freed.

• MallocCorruptionAbort
  Similar to MallocErrorAbort but will not abort in out of memory conditions, making it more useful to catch only those errors which will cause memory corruption. MallocCorruptionAbort is always set on 64-bit processes.

• MallocHelp
  If set, print a list of environment variables that are paid heed to by the allocation-related functions, along with short descriptions. The list should correspond to this documentation.

Note the comments under MallocCorruptionAbort about the behaviour of MallocErrorAbort.

---

For most of my own code, I use a series of wrapper functions — emalloc(), erealloc(), ecalloc(), efree(), estrdup(), etc — that check for failed allocations (efree() is a straight pass-through function for consistency) and do not return when an allocation fails. They either exit or abort. This is basically what Jesus Ramos suggests in his answer; I agree with what he suggests.

However, not all programs can afford to have that happen. I'm just in the process of fixing up some code I wrote which does use these functions so that it can be reused in a context where it is not OK to fail to on allocation error. For its original purpose (security checks during the very early stages of process startup), it was fine to exit on error, but now it needs to be usable after the system is running, when a premature exit is not allowed. So, the code has to deal with those paths where the code used to be able to assume 'no return on allocation failure'. That's a tad painful. It can still take a conservative view; an allocation failure means the request is not safe and process it appropriately. But not all code can afford to fail with abort on memory allocation failure.