Windows XP memory management without pagefile - what are the consequences wrt. to heap fragmentation?

Question

I've been looking into a rather elusive bug that we see in an application on a Windows XP embedded system.

We've narrowed the bug down to a pointer that should be pointing to a block of memory, instead pointing to NULL. Since the memory is allocated with a call to malloc(..) that was going unchecked, my instinct says that the malloc failed and returned NULL (though we are looking for other possibilities right now too, such as race conditions that may inadvertently alter the pointer). This is a native C++ application. The crash was a little more convoluted to track down to this cause, primarily because we only had post-mortem crash dumps and the failure manifested in a third-party library that we don't have source for, on a different thread. Fun times :)

My questions are focused on the memory exhaustion possibility. Of importance is that the XP Embedded system we were running on had its' pagefile disabled.

So, I have three questions; it'd be great if anyone could clarify these for me:

• Primarily, what are the implications of having no paging file? Does this mean that when the heap grows, the new memory needs to be found and allocated immediately by the operating system, even if those free blocks aren't used immediately? I've seen some anecdotal mentions of it, but couldn't find anything specific about exactly what effects disabling the pagefile has.

• Why did Microsoft decide not to enable the Low-Fragmentation Heap by default until Windows Vista? Is there a danger to enabling the LFH for your process on Windows XP?

• What's the difference in WinDbg between 'External Fragmentation' and 'Virtual Address Fragmentation'?

WinDbg reports the heap statistics on the affected heap as follows:

Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast
                  (k)     (k)    (k)     (k) length      blocks cont. heap
04770000 00001002 1621948  94844 1608284  102    6  8068    6      2   L 

  Virtual address fragmentation  94 % (8068 uncommited ranges)

Answer 1

Unlike Linux, on Windows an allocation is a commitment. You will be able to write to the allocated memory. Performance may be horrible, but Windows does not need an OOM killer.

If there's no paging file, a commitment must be backed by RAM. And yet unused memory (such as that used by only during program initialization) is still using up RAM as it can't be paged out. So there's less RAM to make the commitment but a much greater need.

The LFH breaks buggy programs, or better stated: buggy programs may show their bugs in the presence of LFH. Microsoft is extremely friendly towards broken programs, and making LFH opt-in for XP is a typical example of their accommodating behavior.