What does "Private Data" define in VMMAP?

Question

I am using VMMap to analyse Virtual/Process Address Space utilisation in my mixed mode (managed and unmanaged) application. I understand how the Windows VMM and the Virtual Memory API works, I understand how the Heap Memory API Works too. I have looked at the CRT implementation I am using (not in great detail) and (I think I - this could be my downfalling) understand how this uses the aforementioned Win32 APIs.

I'm looking to understand what this "Private Data" stat is showing me. My application makes no direct calls to the any of the Win32 Memory API functions, it only ever uses "malloc/new" in native C++ and "new" in C# (which deep down will be using the Win32 Memory Management API).

The definition of "Private Data" given by VMMap is:

Private memory is memory allocated by VirtualAlloc and not suballocated either by the Heap Manager or the .NET run time. It cannot be shared with other processes, is charged against the system commit limit, and typically contains application data.

So I guess this definition makes me ask, ok, so who is making the calls to VirtualAlloc? Is it the Heap Manager or .Net run time?

I could get an address of some of the committed private data and use WinDbg to find out.... Well... it turns out Microsoft in their wisdom have nobbled the ntdll public symbols, so WinDbg doesn't work so nicely - I can provide more details on this if requested, but basically commands like !address -summary no longer work due to missing symbols.

Another way to word this question might: What C++ or C# code can I write that will cause this private data statistic to increase or decrease? Or is this all managed by the OS, the C++ runtime or the .Net runtime and therefore at the mercy of it's whims?

I can deduce from the nature of VMMap (other memory types are EXCLUSIVE OF EACH OTHER) that this "private data", therefore cannot be any of the following types of address space:

• Heap (note that this includes committed AND reserved heap space - reserved through a call to VirtualAlloc, as described in the description of Private Data above).
• Managed Heap
• Stack
• Shareable
• Mapped File
• Image
• Page Table
• Unusable
• Free

(I couldn't find an online help file that defines what VMMap thinks all the above types are, but here's a link to download the help file: https://technet.microsoft.com/en-us/library/dd535533.aspx)

I have noticed that in my application, the TOTAL (reserved and committed) size of private data remains fairly constant throughout my applications lifetime, despite the Heap/Managed Heap/Stack sizes changing as expected. I have also noticed that of the ~250Mb total used by private data, only ~33Mb is actually committed. Note my method of measuring this is rather rudimentary, so the value may be changing between each of my measurements and I'm just not seeing it (if i knew what this was measuring, I could use DebugDiag to grab a dump of the process when the related counter hit a certain threshold, chicken and egg).

My current speculative theory is that this is space that is being reserved to grow the native (or managed i suppose?) heaps into as they reach their capacity, but I got nothing to prove this. So it remains firmly in the speculative pile.

Searching the internet for details on this can be painful, there are many many posts/articles/blogs out there that confuse things, use self referential definitions (the first sentence of Performance Monitor's definition for Working Set is a great example of this), are incomplete or are just plain wrong. Many places blur definitions or use an inconsistent terminology (note that VMMaps definition of the field private data, goes on to refer to it as private memory, maybe a bit of an anal complaint, but ambiguity).

Now that i've criticised the rest of the internet for getting things muddled and incorrect... if there is anything in the above that doesn't make sense or you can show me documentation to the contrary or you need a more explicit definition of, let me know and i'll put myself on the offenders list as well! I think the first half of trying to explain a memory issue to someone, online, is making sure we're all talking about the same thing.

Finally this question: How does VMMap know a given memory region is Thread Stack, specifically? suggests I may never find out an answer :/

UPDATE/EDIT: I have discovered that by turning on gflags user stack tracing (gflags -i myapp.exe +ust), you can increase the size of the Private Data, I would assume this is the backtrace database, but there even without gflags, there is still Private Data which I am struggling to account for.

Answer 1

I know it's a pretty old question. But it's still unanswered for some reason. This question was mentioned by Sasha Goldstein in his talk about WinDbg at DotNext conference - https://www.youtube.com/watch?v=8t1aTbnZ2CE. The point was it can be easily answered with help of WinDbg.

To answer whether CLR uses VirtualAlloc for its heap we'll set a breakpoint at this function with a script which prints current stack (native and managed).

bp kernelbase!VirtualAlloc ".printf \"allocating %d bytes of virtual memory\", dwo(@esp+8);.echo;  k 5; !clrstack; gc"

Here: k 5 print last 5 frames of native callstack and !clrstack (from SOS) prints managed stack. gc continue execution.

Please note that this script will work only for x86 processes. For x64 you'll need some other (registries and calling convention differ).

Then I created a simple program which allocate an object and adds it to a List.

    static void Main(string[] args)
    {
        var list = new List<string[]>();
        while (true) {
            var a = Console.ReadLine();
            if (a == "q" || a == "Q") break;
            var arr = new string[100];
            list.Add(arr);
        }
    }

Run it under WinDbg and started pressing Enter. At some point the breakpoint hit - on List expanding and allocating additional memory in the heap:

So obviously CLR uses VirtualAlloc for allocating memory for its heap.