Why does malloc not work sometimes?

Question

I'm porting a C project from Linux to Windows. On Linux it is completely stable. On Windows, it's working well most times, but sometimes I got a segmentation fault.

I'm using Microsoft Visual Studio 2010 to compile and debug and looks like sometimes my malloc calls simply doesn't allocate memory, returning NULL. The machine has free memory; it already passed through that code a thousand times, but it still happens in different locations.

Like I said, it doesn't happen all the time or in the same location; it looks like a random error.

Is there something I have to be more careful on Windows than on Linux? What can I be doing wrong?

Answer 1

malloc() returns an invalid pointer of NULL when it is unable to service a memory request. In most cases the C memory allocation routines manage a list or heap of memory available memory with calls to the operating system to allocate additional chunks of memory when a malloc() call is made and there is not a block on the list or heap to satisfy the request.

So the first case of malloc() failing is when a memory request can not be satisfied because (1) there is not a usable block of memory on the list or heap of the C runtime and (2) when the C runtime memory management requested more memory from the operating system, the request was refused.

Here is an article about Pointer Allocation Strategies.

This forum article gives an example of malloc failure due to memory fragmentation.

Another reason why malloc() might fail is because the memory management data structures have become corrupted probably due to a buffer overflow in which a memory area that was allocated was used for an object larger than the size of the memory allocated. Different versions of malloc() can use different strategies for memory management and determining how much memory to provide when malloc() is called. For instance a malloc() may give you exactly the number of bytes requested or it may give you more than you asked for in order to fit the block allocated within memory boundaries or to make the memory management easier.

With modern operating systems and virtual memory, it is pretty difficult to run out of memory unless you are doing some really large memory resident storage. However as user Yeow_Meng mentioned in a comment below, if you are doing arithmetic to determine the size to allocate and the result is a negative number you could end up requesting a huge amount of memory because the argument to malloc() for the amount of memory to allocation is unsigned.

You can run into the problem of negative sizes when doing pointer arithmetic to determine how much space is needed for some data. This kind of error is common for text parsing that is done on text that is unexpected. For example the following code would result in a very large malloc() request.

char pathText[64] = "./dir/prefix";  // a buffer of text with path using dot (.) for current dir
char *pFile = strrchr (pathText, '/');  // find last slash where the file name begins
char *pExt = strrchr (pathText, '.');    // looking for file extension 

// at this point the programmer expected that
//   - pFile points to the last slash in the path name
//   - pExt point to the dot (.) in the file extension or NULL
// however with this data we instead have the following pointers because rather than
// an absolute path, it is a relative path
//   - pFile points to the last slash in the path name
//   - pExt point to the first dot (.) in the path name as there is no file extension
// the result is that rather than a non-NULL pExt value being larger than pFile,
// it is instead smaller for this specific data.
char *pNameNoExt;
if (pExt) {  // this really should be if (pExt && pFile < pExt) {
    // extension specified so allocate space just for the name, no extension
    // allocate space for just the file name without the extension
    // since pExt is less than pFile, we get a negative value which then becomes
    // a really huge unsigned value.
    pNameNoExt = malloc ((pExt - pFile + 1) * sizeof(char));
} else {
    pNameNoExt = malloc ((strlen(pFile) + 1) * sizeof(char));
}

A good run time memory management will try to coalesce freed chunks of memory so that many smaller blocks will be combined into larger blocks as they are freed. This combining of chunks of memory reduces the chances of being unable to service a memory request using what is already available on the list or heap of memory being managed by the C memory management run time.

The more that you can just reuse already allocated memory and the less you depend on malloc() and free() the better. If you are not doing a malloc() then it is difficult for it to fail.

The more that you can change many small size calls to malloc() to fewer large calls to malloc() the less chance you have for fragmenting the memory and expanding the size of the memory list or heap with lots of small blocks that can not be combined because they are not next to each other.

The more that you can malloc() and free() contiguous blocks at the same time, the more likely that the memory management run time can coalesce blocks.

There is no rule that says you must do a malloc() with the specific size of an object, the size argument provided to malloc() can be larger than the size needed for the object for which you are allocating memory. So you may want to use some kind of a rule for calls to malloc () so that standard sized blocks are allocated by rounding up to some standard amount of memory. So you may allocate in blocks of 16 bytes using a formula like ((size / 16) + 1) * 16 or more likely ((size >> 4) + 1) << 4. Many script languages use something similar so as to increase the chance of repeated calls to malloc() and free() being able to match up a request with a free block on the list or heap of memory.

Here is a somewhat simple example of trying to reduce the number of blocks allocated and deallocated. Lets say that we have a linked list of variable sized blocks of memory. So the struct for the nodes in the linked list look something like:

typedef struct __MyNodeStruct {
    struct __MyNodeStruct *pNext;
    unsigned char *pMegaBuffer;
} MyNodeStruct;

There could be two ways of allocating this memory for a particular buffer and its node. The first is a standard allocation of the node followed by an allocation of the buffer as in the following.

MyNodeStruct *pNewNode = malloc(sizeof(MyNodeStruct));
if (pNewNode)
    pNewNode->pMegaBuffer = malloc(15000);

However another way would be to do something like the following which uses a single memory allocation with pointer arithmetic so that a single malloc() provides both memory areas.

MyNodeStruct *pNewNode = malloc(sizeof(myNodeStruct) + 15000);
if (pNewNode)
    pNewNode->pMegaBuffer = ((unsigned char *)pNewNode) + sizeof(myNodeStruct);

However if you are using this single allocation method, you will need to make sure that you are consistent in the use of the pointer pMegaBuffer that you do not accidently do a free() on it. And if you are having to change out the buffer with a larger buffer, you will need to free the node and reallocate buffer and node. So there is more work for the programmer.

Answer 2

Another reason for malloc() to fail on Windows is if your code allocates in one DLL and deallocates in a different DLL or EXE.

Unlike Linux, in Windows a DLL or EXE has its own links to the runtime libraries. That means that you can link your program, using the 2013 CRT to a DLL compiled against the 2008 CRT.

The different runtimes might handle the heap differently. The Debug and Release CRTs definitely handle the heap differently. If you malloc() in Debug and free() in Release, it will break horribly, and this might be causing your problem.