Task Manager disagrees with Process Explorer?

Question

I'm trying to track down a memory leak in my .NET app. Windows Task Manager reports that the memory usage stays constant, while Process Explorer reports that the memory usage is raising.

In Task Manager, I'm looking in the only memory column, "Memory (Private working set)". In Process Explorer I'm looking in the "Private bytes" column because it's raising and the value under "Working set" isn't.

Now, certainly Process Explorer is the right one because after several allocations, my app crashes with an Out of memory exception. The question is, why does Task Manager misreport the memory usage of the app? Not only that, but it also misreports the global system free memory (the graph in the Performance tab stays constant).

My code shouldn't be needed, but here it is for completeness. It shows an empty window that holds a big array. When any key is pressed, the window is closed and a new one is opened, holding a new array. The old window has leaked, probably due to a bug in the qt4dotnet GUI library.

using System;
using com.trolltech.qt.gui;

namespace LeakTest
{
    class Test : QWidget
    {
        public byte[] Data = new byte[1000 * 1000 * 100];

        public Test()
        {
            show();
            GC.Collect(); // so measurements are more accurate
        }

        protected override void keyPressEvent(QKeyEvent arg__1)
        {
            disposeLater();
            new Test();
        }

        [STAThread]
        static void Main(string[] args)
        {
            QApplication.initialize(args);

            new Test();

            QApplication.exec();
        }
    }
}

OS: Windows 7

Interesting note: when I make "Data" a 2D jagged array of dimensions [1000 * 1000 * 100][1], Task Manager does report raising memory usage.

Answer 1

They are two completely different memory measures. Working set is the amount of RAM your program uses. It is a constantly changing number and affected by how much RAM other processes need. You can't run out of RAM, Windows makes RAM available to you by swapping mapped pages out to the paging file.

Private bytes is the amount of virtual memory your program uses that isn't shared with any other process. On a 32-bit machine, you have 2 gigabytes of virtual memory available. It needs to be shared between code and data. You get OOM when there isn't enough space left in the addressable amount of virtual memory to fit the requested allocation. Yes, it is the more accurate number.

Asking for 100 megabytes at a time is risky. Virtual memory space can get fragmented and after a while there might still be lots of free virtual memory but not a hole that large enough to fit 100 megabytes. A jagged array solves this problem because it is an array of arrays, the required chunks are much smaller so can easily fit whatever holes are left.

A 64-bit operating system completely solves this problem. Your program has many gigabytes of address space available, in practice limited only by the maximum size of the paging file. You simply can't run out of big holes.

Answer 2

If you want to track down a memory leak, and you can't find it by reading the code, then you probably need a real tool that is designed for the job.

Neither Task Manager nor Process Explorer are appropriate tools for debugging memory leaks – you want something that will tell you where you allocate the memory that is not being returned.