valgrind output in C++ of "still reachable" and "possibly lost" blocks do not reference my sources

Question

I am having a hard time pinpointing where do I get memory leaks in my code.

The valgrind command I run:

valgrind --leak-check=full --log-file=vg1.log --show-leak-kinds=all --leak-resolution=low --track-origins=yes --leak-check-heuristics=all ./enalu_dbg

and the output

==22866== Memcheck, a memory error detector
==22866== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==22866== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==22866== Command: ./enalu_dbg
==22866== Parent PID: 21933
==22866== 
==22866== 
==22866== HEAP SUMMARY:
==22866==     in use at exit: 47,252 bytes in 240 blocks
==22866==   total heap usage: 288 allocs, 48 frees, 55,138 bytes allocated
==22866== 
==22866== 4 bytes in 1 blocks are still reachable in loss record 1 of 23
==22866==    at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==22866==    by 0x77018CD: ??? (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4002.0)
==22866==    by 0x7701D28: g_private_get (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4002.0)
==22866==    by 0x76DB20C: g_slice_alloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4002.0)
==22866==    by 0x76AF17D: g_hash_table_new_full (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4002.0)
==22866==    by 0x76CF494: g_quark_from_static_string (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4002.0)
==22866==    by 0x74314AB: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.4002.0)
==22866==    by 0x4010139: call_init.part.0 (dl-init.c:78)
==22866==    by 0x4010222: call_init (dl-init.c:36)
==22866==    by 0x4010222: _dl_init (dl-init.c:126)
==22866==    by 0x4001309: ??? (in /lib/x86_64-linux-gnu/ld-2.19.so)

...

==22866== 184 bytes in 1 blocks are possibly lost in loss record 13 of 23
==22866==    at 0x4C2CE8E: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==22866==    by 0x76C56AE: g_realloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4002.0)
==22866==    by 0x7451618: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.4002.0)
==22866==    by 0x74560D4: g_type_register_static (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.4002.0)
==22866==    by 0x7442DE6: g_param_type_register_static (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.4002.0)
==22866==    by 0x74449AB: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.4002.0)
==22866==    by 0x74315E9: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.4002.0)
==22866==    by 0x4010139: call_init.part.0 (dl-init.c:78)
==22866==    by 0x4010222: call_init (dl-init.c:36)
==22866==    by 0x4010222: _dl_init (dl-init.c:126)
==22866==    by 0x4001309: ??? (in /lib/x86_64-linux-gnu/ld-2.19.so)
==22866== 

...

==22866== 6,028 bytes in 60 blocks are still reachable in loss record 21 of 23
==22866==    at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==22866==    by 0x76C5668: g_malloc0 (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4002.0)
==22866==    by 0x74514D9: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.4002.0)
==22866==    by 0x74560D4: g_type_register_static (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.4002.0)
==22866==    by 0x7442DE6: g_param_type_register_static (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.4002.0)
==22866==    by 0x744423A: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.4002.0)
==22866==    by 0x74315E9: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.4002.0)
==22866==    by 0x4010139: call_init.part.0 (dl-init.c:78)
==22866==    by 0x4010222: call_init (dl-init.c:36)
==22866==    by 0x4010222: _dl_init (dl-init.c:126)
==22866==    by 0x4001309: ??? (in /lib/x86_64-linux-gnu/ld-2.19.so)
==22866== 
==22866== 10,360 bytes in 5 blocks are still reachable in loss record 22 of 23
==22866==    at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==22866==    by 0x8B16E9A: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.30.2)
==22866==    by 0x8B15ACE: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.30.2)
==22866==    by 0x8B17585: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.30.2)
==22866==    by 0x8AC9508: ??? (in /usr/lib/x86_64-linux-gnu/libpixman-1.so.0.30.2)
==22866==    by 0x4010139: call_init.part.0 (dl-init.c:78)
==22866==    by 0x4010222: call_init (dl-init.c:36)
==22866==    by 0x4010222: _dl_init (dl-init.c:126)
==22866==    by 0x4001309: ??? (in /lib/x86_64-linux-gnu/ld-2.19.so)
==22866== 
==22866== 16,600 bytes in 4 blocks are still reachable in loss record 23 of 23
==22866==    at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==22866==    by 0x76C5610: g_malloc (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4002.0)
==22866==    by 0x76CF445: g_quark_from_static_string (in /lib/x86_64-linux-gnu/libglib-2.0.so.0.4002.0)
==22866==    by 0x74314AB: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.4002.0)
==22866==    by 0x4010139: call_init.part.0 (dl-init.c:78)
==22866==    by 0x4010222: call_init (dl-init.c:36)
==22866==    by 0x4010222: _dl_init (dl-init.c:126)
==22866==    by 0x4001309: ??? (in /lib/x86_64-linux-gnu/ld-2.19.so)
==22866== 
==22866== LEAK SUMMARY:
==22866==    definitely lost: 0 bytes in 0 blocks
==22866==    indirectly lost: 0 bytes in 0 blocks
==22866==      possibly lost: 1,352 bytes in 18 blocks
==22866==    still reachable: 45,900 bytes in 222 blocks
==22866==                       of which reachable via heuristic:
==22866==                         newarray           : 1,536 bytes in 16 blocks
==22866==         suppressed: 0 bytes in 0 blocks
==22866== 
==22866== For counts of detected and suppressed errors, rerun with: -v
==22866== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)

Most of the records shown (but 1) are "still reachable blocks". I have read that these may be due to delayed pool deallocation. ie deallocations of containers (such as vectors , of which I use alot) after the termination of my main(argc,argv){} function.

However, there is surely something wrong because after about 8-9 hours of execution, I clearly see that the executable is using more memory than it started of with (in my pc it started with 0.6% memory usage and after 8h it used about 6%).

The problem is that these are cryptic messages - and absolutely none originates from my source files. AS I read here "_dl*" calls are related to linux loader. So how can I pinpoint where the problem originates?

I should add that this code uses

1. 1+3 threads(for all blocking operations, ie. to read from stdin and serial port and write data to a file),
2. the boost library (especially circular buffers), and
3. the gsl library.

I have however built the code from small proof of concept parts, which do not show any errors/warnings in valgrind.

Additionally, I have only a limited number of pointers to class objects in my code, which I verified I delete in the respective destructors.

Answer 1

The original allocation calls are from the shared library initialization code, that gets executed when the shared library your application links with is loaded at runtime, typically before any of your application code actually runs. This is why you don't see your code in the backtrace. It didn't even run yet.

The key symbol to look for is _dl_init, the entry point for the shared library initialization. Looking upstream of that tells you which library is being initialized. In your case, it's a bunch of Gnome libraries, and a library called "libpixman".

Shared libraries also have a cleanup function that gets invoked when the shared library gets unloaded.

A well-organized shared library will use the shared library cleanup function to orderly deallocate all memory that it allocated at startup. Unfortunately, this kind of inattentiveness to detail is sadly common: shared libraries allocating a bunch of memory from the heap, for the shared library's internal static tables, without bothering to deallocate that memory when the shared library gets unloaded.

It's unlikely that this is responsible for the memory leaks you're observing while your application is running, except in one case which I'll mention in a moment. It's been my experience that this sloppy allocation practice is only used for static tables that shared libraries allocate once, when they get loaded. The thinking that goes here is that it's not necessary to explicitly clean up after oneself, because the library will only get unloaded once, when the process exits.

Sadly, the developers that cut these kind of corners have never heard of dlopen() and dlclose(). This makes it impossible for large applications to load the shared library on demand only when it's needed and unload it afterwards, until it's needed again.

So, unless your application code is repeatedly dlopen()ing and dlclose()ing all these Gnome libraries, and libpixman, you'll have to continue looking for your leaks somewhere else. You should read up on, and employ valgrind's suppression files, to suppress this annoying noise from its output.