Need help in reading callgrind output

Question

I have run callgrind with my application like this:

valgrind --tool=callgrind MyApplication

and then call:

callgrind_annotate --auto=yes ./callgrind.out.2489

I see output like:

 768,097,560  PROGRAM TOTALS

--------------------------------------------------------------------------------
       Ir  file:function
--------------------------------------------------------------------------------
18,624,794  /build/buildd/eglibc-2.11.1/elf/dl-lookup.c:do_lookup_x
[/lib/ld-2.11.1.so]
18,149,492  /src/js/src/jsgc.cpp:JS_CallTracer'2
[/src/firefox-debug-objdir/js/src/libmozjs.so]
16,328,897 /src/layout/style/nsCSSDataBlock.cpp:nsCSSExpandedDataBlock::DoAssertInitialState()
[/src/firefox-debug-objdir/toolkit/library/libxul.so]
13,376,634  /build/buildd/eglibc-2.11.1/nptl/pthread_getspecific.c:pthread_getspecific
[/lib/libpthread-2.11.1.so]
13,005,623  /build/buildd/eglibc-2.11.1/malloc/malloc.c:_int_malloc
[/lib/libc-2.11.1.so]
10,404,453  ???:0x0000000000009190 [/usr/lib/libpangocairo-1.0.so.0.2800.0]
10,358,646  /src/xpcom/io/nsFastLoadFile.cpp:NS_AccumulateFastLoadChecksum(unsigned
int*, unsigned char const*, unsigned int, int)
[/src/firefox-debug-objdir/toolkit/library/libxul.so]
 8,543,634  /src/js/src/jsscan.cpp:js_GetToken
[/src/firefox-debug-objdir/js/src/libmozjs.so]
 7,451,273  /src/xpcom/typelib/xpt/src/xpt_arena.c:XPT_ArenaMalloc
[/src/firefox-debug-objdir/toolkit/library/libxul.so]
 7,335,131  ???:g_type_check_instance_is_a [/usr/lib/libgobject-2.0.so.0.2400.0]

I have a few questions:

1. What does the number on the right mean? Does it mean it spend accumulative that long in calling the function on the right? How can I tell how many times that function has been called and Does that include the time spend in calling the functions called by that function?

2. What does line with ??? mean? e.g. ???:0x0000000000009190 [/usr/lib/libpangocairo-1.0.so.0.2800.0]

Answer 1

Use KCachegrind. Deciphering the text output is just meaningless.

Answer 2

As Let_Me_Be already answered, KCachegrind is the preferred way to go. Also make sure the dot command is available on your system in order to generate graphs with it. There's also the callgrind_annotate tool, which can do some basic processing at the command-line level.

Regarding your second question, these are calls inside libraries without debugging information. Usually it's not that interesting, but if you really need that information, you should compile the library yourself with debugging symbols (and optimization flags, since you're profiling).