C API design: what to do when malloc returns NULL?

Question

Let's say I'm writing a little library in C -- some data structure, say. What should I do if I'm unable to allocate memory?

It might be pretty important, e.g. I need some memory to initialize the data structure in the first place, or I'm inserting a key-value pair and want to wrap it in a little struct. It could also be less critical, for instance something like a pretty_print function that builds up a nice string representation of the contents. However, it's typically more serious than your average error -- there might not be a point in continuing at all. A ton of sample uses of malloc online just straight up exit the program if it returns NULL. I'm guessing a lot of real client code does that too -- just pop up some error, or write it to stderr, and abort. (And a lot of real code probably doesn't check the return value of malloc at all.)

Sometimes it makes sense to return NULL, but not always. Error codes (or just some boolean success value), either as return values or out parameters work fine, but it seems like they can clutter up or hurt the readability of the API (then again, maybe that's somewhat expected in a language like C?). Another option is to have some sort of internal error state the caller can subsequently query, e.g. with a get_error function, but then you have to be careful about thread safety, and it might be easy to miss; people tend to be lax about checking for errors anyway, and if it's a separate function altogether they might not know about it, or they might not bother (but then I guess that's their problem).

(I've sometimes seen malloc wrapped in a function that just tries again until memory is available...

void *my_malloc(size_t size) {     void *result = NULL;     while (result == NULL)         result = malloc(size);     return result; } 

But that seems kind of silly and maybe dangerous.)

What's a proper way to handle this?

Answer 1

If allocation fails in a way that prevents forward progress, the only acceptable solution for library code is to back-out whatever allocations and other changes have already been made in the partially-completed operation and return a failure code to the caller. Only the calling application can know what the right way to proceed is. Some examples:

1. A music player might just abort or return to an initial/stopped state and wait for user input again.
2. A word processor might need to store an emergency dump of the current document state to a recovery file then abort.
3. A high-end database server might need to reject and back out the whole transaction and report to the client.

If you follow the oft-advised but backwards idea that your library should just abort the caller on allocations failures, you will either have many programs that determine they cannot use your library for this reason, your users' of the programs that use your library will be extremely angry when an allocation failure causes their valuable data to be thrown away.

Edit: One objection some of the "abort" camp will raise against my answer is that, on systems with overcommit, even calls to malloc that appeared to succeed may fail when the kernel tries to instantiate physical storage for the virtual memory allocated. This ignores the fact that anyone needing high reliability will have overcommit disabled, as well as the fact that (at least on 32-bit systems) allocation failure is more likely due to virtual address space exhaustion than physical storage exhaustion.

Answer 2

Simply return an error in whatever way you normally do. Since we're talking about an API, you don't know what environment you're being called from, so just return NULL or follow whatever other error-handling procedure you already use. You don't want to loop forever, since the caller may not really need that memory and they would rather just know that you can't handle it, or perhaps the caller has a user-interface that they can send the error to.

Most APIs will have some sort of return value that indicates an error across all functions, other APIs require that the caller call a special "check_error" function to determine if there is an error. You may also want a "get_error" function to return an error string that the caller can optionally display to the user or include in a log. It should be descriptive: "so-and-so API has encountered an error in function whatever: unable to allocate memory". Or whatever. Enough that when someone gets the error, they know what component threw it, and when he emails you with the log message, you know exactly what went wrong.

Of course you can also just crash, but that prevents the caller from closing out anything else they may have been doing and looks ugly, if your code has a habit of dying when it is called rather than returning an error, people are going to look for a library that doesn't actively try to kill their program.