Resident memory increase while valgrind not showing any leaks

Question

I have added an option of restarting in my C C++ programme code. Every time it restarts I can see increase in Resident memory, while valgrind is not showing any leak. What can be the reason of resident memory increase.

Answer 1

There are several possible reasons for this:

1. You may be growing memory, but not actually leaking: Something like vector<int> v; for(;;) v.push_back(1); will run out of memory within a few seconds, but is not a leak according to valgrind.
2. The heap is part of your "res" memory, so if you have something that allocates x MB of heap memory, then releases it, unless the OS actually need that memory for other purposes, it will remain as part of your applications memory. (Actually, it's quite a bit more complex than that, but for this discussion, this picture is valid).
3. Heap fragmentation. If you allocate and free things in the heap, and the memory size varies a lot, the memory is indeed "free", but the lumps in the heap may be too small to use for the next allocation, so a "fresh" piece of heap is cut up. Imagine that you start out with a long piece of wood, and cut it down to fit somewhere, then remove it and cut it again - you can never make it back up to the full size again. (Again, in reality, it's a lot more complex than this, but for this discussion it's sufficient).

You may want to use the massif tool for valgrind, valgrind --tool=massif prog to identify things like #1, where data is building up over time.

Edit: From the page in the link:

Massif is a heap profiler. It measures how much heap memory your program uses. This includes both the useful space, and the extra bytes allocated for book-keeping and alignment purposes. It can also measure the size of your program's stack(s), although it does not do so by default.

The mem_heap_extra_B is the number of bytes allocated as "padding" and "overhead". It will typically be a small portion of the total allocation size, but if you have many very small allocations, it can dominate the heap usage.

The mem_stacks_B is the number of bytes of stack the application uses (by default, this is not measured, as it slows down the code dramatically).

Take this program as an example:

#include <vector>

int main(int argc, char **argv)
{
    (void)argv;   // Not used. 

    const int size = 10000;

    std::vector<int*> v(size);

    switch (argc)
    {
    case 1:
        for(int i = 0; i < size; i++)
        {
            v[i] = new int;
        };

        for(int i = 0; i < size; i++)
        {
            delete v[i];
        };
        break;

    case 2:
    {
        int *t = new int [size];
        for(int i = 0; i < size; i++)
        {
            v[i] = t+i;
        };

        delete [] t;
        break;
    }

    }

    return 0;
}

The following is the output from valgrind --tool=massif ./a.out (so case 1: variant of the code) (at the peak usage):

time=2872338
mem_heap_B=120000
mem_heap_extra_B=200008
mem_stacks_B=0
heap_tree=peak

where if we run with valgrind --tool=massif ./a.out 1 (so case 2: criant of the code), at peak usage, the output is this:

time=2523909
mem_heap_B=120000
mem_heap_extra_B=16
mem_stacks_B=0
heap_tree=peak

Note how different the mem_heap_extra_B is - in the first case, it's greater than the actual heap usage (because every 4 byte allocating actually takes up a lot more), where in the second case, the extra bytes is only 16, but the actual "used" size of the heap is the same, 120000 (which makes sense, we have 10000 pointers at 8 bytes, + 10000 integers at 4 bytes each).

Unfortunately, this program is rather uninteresting when it comes to stack usage, but if you use --stacks=yes, it will show the number of bytes of stack used, e.g.:

time=2222719
mem_heap_B=120000
mem_heap_extra_B=16
mem_stacks_B=528

If I grep for mem_stacks_B in the Pascal compiler I'm working on, then it gets more interesting:

 10:mem_stacks_B=0
 18:mem_stacks_B=1576
 26:mem_stacks_B=1368
 34:mem_stacks_B=1368
 42:mem_stacks_B=1576
 50:mem_stacks_B=120
 58:mem_stacks_B=2592
 66:mem_stacks_B=4656
288:mem_stacks_B=2464
296:mem_stacks_B=43104
431:mem_stacks_B=2424
439:mem_stacks_B=10960
447:mem_stacks_B=8096
622:mem_stacks_B=8352
887:mem_stacks_B=3816
895:mem_stacks_B=3360
903:mem_stacks_B=3664
911:mem_stacks_B=3216

There is a lot more, but sufficient to show that "stack usage varies quite a bit".

As you can see, it varies quite a bit. No, I haven't tried to figure out what it's doing when it's using 43KB of stack - it's not really outrageous anyway.