Help with strange memory behavior. Looking for leaks both in my brain and in my code

Question

I spent the last few days trying to find memory leaks in a program we are developing.

First of all, I tried using some leak detectors. After fixing a few issues, they do not find any leaks any more. However, I am also monitoring my application using perfmon.exe. Performance Monitor reports that 'Private Bytes' and 'Working Set - Private' are steadily rising when the app is used. To me, this suggests that the program is using more and more memory the longer it runs. Internal resources seem to be stable however, so this sounds like leaking to me.

The program is loading a DLL at runtime. I suspect that these leaks or whatever they are occur in that library and get purged when the library is unloaded, hence they won't get picked up by the leak detectors. I used both DevPartner BoundsChecker and Virtual Leak Detector to look for memory leaks. Both supposedly catch leaks in DLLs.

Also, the memory consumption is increasing in steps and those steps roughly, but not exactly, coincide with certain GUI actions I perform in the application. If these were errors in our code, they should get triggered every single time the actions are performed and not just most of the time.

Whenever I am confronted with so much strangeness, I begin to question my basic assumptions. So I turn to you, who know everything, for suggestions. Is there a flaw in my assumptions? Do you have an idea of how to go about troubleshooting a problem like this?

Edit:
I am currently using Microsoft Visual C++ (x86) on Windows 7 64.

Edit2:
I just used IBM Purify to hunt for leaks. First of all, it lists a full 30% of the program as leaked memory. This can not be true. I guess it is identifying the whole DLL as leaked or something like that. However, if I search for new leaks every few actions, it reports leaks that correspond with the size increase reported by Performance Monitor. This could be a lead to a leak. Sadly, I am only using the trial version of Purify, so it won't show me the actual location of those leaks. (These leaks only show up at runtime. When the program exits, there are no leaks whatsoever reported by any tool.)

Answer 1

Monitoring the app's memory use with PerfMon or Task Manager is not a valid way of checking for memory leaks. For example, your runtime may just be holding on to extra memory from the OS for pre-allocation purposes or due to fragmentation.

The trick, in my experience, is the CRT debug heap. You can request information about all live objects, and the CRT provides functions to compare snapshots.

http://msdn.microsoft.com/en-us/library/wc28wkas.aspx

Answer 2

It's difficult to know without seeing your code, but there are less obvious ways that "leaks" can occur in a C++ program, e.g.

1. memory fragmentation - if you are allocating different size objects all the time, then sometimes there won't be a large enough contiguous area of free memory and more will have to be allocated from the OS. Allocations like this will not be freed back to OS until all the memory in the allocation is freed, so long running programs will tend to grow (in terms of address space used) over time.

2. forgetting to have a virtual in a base case which has virtual functions - a very common gotcha which leads to leaks.

3. using smart pointers, such as shared_ptr, and have an object hold on to a shared_ptr to an object - memory leak tools won't usually spot this kind of thing.

4. using smart pointers and getting circular references - you need to use e.g. a weak_ptr somewhere to break the cycle.

As to tools, there is purify which is good but expensive.