Can reserved memory cause an Out Of memory exception

Question

We have a 32 bit windows service that leaks memory - OutOfMemory exception is thrown. It is .net 4.0 executable running on windows server 2003. While debugging crash dump files using WinDbg, I see that most of the memory is actually reserved and not commited.

As you can see from WinDbg screenshot, there is 2.5 Gb of unclassified memory usage, and most of it 2.1 Gb is actually reserved memory (MEM_RESERVE). I have experience debugging crush dumps, but this scenario is something new to me. MEM_COMMIT is quit OK - 564.270 Mb, managed heap's size is abt 82 Mb

I also checked native heaps to see, if there are big chunks of data preserved, but couldn't find anything suspicious there either

So my question is - is it possible that MEM_RESERVED might result in OOM exception? If so, how can I debug it, see why/how huge amount of memory is being reserved? Where else would you look to find what might be the problem?

If any other information is required please ask for it, and I will update my post.

Answer 1

Yes, reserving memory can trigger OutOfMemoryException. Try allocating a couple of very large byte arrays. The memory for these will not be committed until you write to the content of the arrays. However, you can easily trigger OOM just by allocating these arrays.

I don't know the implementation details, but since VirtualAlloc will fail if it cannot honor a reserve request, I assume the CLR translates this into an exception. I don't see how it could turn a failed reserve request into anything useful, so an exception is a sensible choice.

Answer 2

OutOfMemoryException occurs if the system cannot allocate more virtual memory for your application. Reserved memory is virtual memory and therefore accounts to that limit.

You can try that in C++ most easily:

while(::VirtualAlloc(NULL, 65536, MEM_RESERVE, PAGE_READWRITE) != NULL );
std::cout << "All memory reserved. Now check with tools." << std::endl;

If VirtualAlloc() returns NULL, it could not allocate more memoy. In WinDbg this will show

MEM_RESERVE   32307  7f6f2000 (1.991 Gb)  99.59%   99.56%

However, VirtualAlloc() does not allocate memory on the heap, so !heap is not useful in this case and shows only the default process heap:

0:000> !heap
Index   Address  Name      Debugging options enabled
  1:   00440000 

It's the other way round: the heap mamanger uses VirtualAlloc() to get the memory. Also note that .NET does not use the heap manager as well. It also allocates memory directly using VirtualAlloc() and then manages it on its own. So, because you can see it in the output of !heap it's not a .NET issue, it's a native memory issue.

In my naive understanding the gflags setting Enable heap tagging by DLL should be helpful to determine the source of a heap allocation. However, my expectation that the command !heap -t would simply display the name of the DLL which allocated the memory did not come true.