Realloc() does not correctly free memory in Windows

Question

I am attempting to use realloc() in a Windows application. I am allocating a large block of memory, then using realloc() to shrink it down later once I know the correct size.

I am finding that although realloc() appears to work correctly (memory in Task Manager reflects what you would expect) the application eventually runs out of memory. From what I can tell, it's as though relloc() frees the memory but does not free the virtual address space associated with the memory. As a result, malloc() will eventually fail.

Here's a small console app that demonstrates the problem:

int _tmain(int argc, _TCHAR* argv[])
{
    static const DWORD dwAllocSize = (50 * 1024 * 1024);
    static const DWORD dwReallocSize = 10240;
    static const DWORD dwMaxIterations = 200;

    BYTE* arpMemory[dwMaxIterations];
    memset( arpMemory, 0, sizeof(arpMemory) );

    for( DWORD i = 0; i < dwMaxIterations; i++ )
    {
        arpMemory[i] = (BYTE*) malloc( dwAllocSize );
        if( !arpMemory[i] )
        {
            printf("OUT OF MEMORY after %d iterations!\n", i);
            return -1;
        }

        BYTE* pRealloc = (BYTE*) realloc( arpMemory[i], dwReallocSize );
        if( !pRealloc )
        {
            printf("Realloc FAILED after %d iterations!\n", i);
            return -1;
        }
        else if( pRealloc != arpMemory[i] )
        {
            printf("Warning: Pointer changed: 0x%08X -> 0x%08X\n", arpMemory[i], pRealloc);
            arpMemory[i] = pRealloc;
        }
    }

    printf("Success!\n");

    for( int i = 0; i < dwMaxIterations; i++ )
        free( arpMemory[i] );

    return 0;
}

The app repeatedly allocates 50 MB of memory and then immediately resizes it to be only 10K. If you run it, you will find that it fails with an OUT OF MEMORY error after only 38 iterations. This corresponds to 2GB of originally allocated memory -- which is the address space limit for Windows applications.

Interestingly, if you look in Task Manager, you will see the application taking almost no memory at all. Yet malloc() is failing. This is what leads me to believe the virtual address space is being exhausted.

(Another experiment to try is to comment out the reallocation, so no memory is freed or reallocated. The app fails in exactly the same place: After 38 iterations. The only difference is that this time Task Manager reflects the full 2GB in use.)

One final point of information: This same application works under Linux. So this realloc() problem is strictly Windows-only.

Any thoughts?

Answer 1

You are fragmenting the heap doing this. Whatever you release with realloc() gets added to the list of free blocks. Never to be used again because you always ask for a new block that's larger than that. These free blocks will just accumulate taking up virtual memory until no more is left. Happens pretty quickly when you throw away almost 50 megabytes at a time.

You'll need to rethink your approach.