Is there an equivalent of set_new_handler() for malloc() failures?

Question

In C++, you can arrange for a function to be called whenever new fails. Is there a way to have a function called whenever malloc fails? Assume that malloc is being called from third-party libraries that I don't want to change.

I don't think there's a portable answer, so I'll happily accept platform-specific ones. I'm using Linux+uclibc on some platforms and Linux+glibc on others. I'm planning to use setrlimit to limit the amount of memory malloc can return.

Answer 1

malloc returns NULL if it fails. You should be handling that, and other failures from CRT memory functions (realloc especially is easy to get wrong).

In the general case, I think you'd have to wrap all CRT usage of memory in functions of your own devising to redirect on error.

On Windows you can hook into the CRT using Allocation Hook Functions, that might be a way to do what you want. This gives you a hook for handling CRT calls via logic for onalloc, onrealloc, onfree, effectively.

I make no guarantees since I am a Windows guy but it looks like malloc_hook on Linux offers the same as that Windows hook allows. These methods should enable you to capture all CRT memory calls without changing code in the third-party libraries, assuming they all use the same CRT at runtime - always a good idea, but not guaranteed on Windows at least...