How do you get debugging symbols working in linux perf tool inside Docker containers?

Question

I am using Docker containers based on the "ubuntu" tag and cannot get linux perf tool to display debugging symbols.

Here is what I'm doing to demonstrate the problem.

First I start a container, here with an interactive shell.

$ docker run -t -i ubuntu:14.04 /bin/bash

Then from the container prompt I install linux perf tool.

$ apt-get update
$ apt-get install -y linux-tools-common linux-tools-generic linux-tools-`uname -r`

I can now use the perf tool. My kernel is 3.16.0-77-generic.

Now I'll install gcc, compile a test program, and try to run it under perf record.

$ apt-get install -y gcc

I paste in the test program into test.c:

#include <stdio.h>

int function(int i) {
    int j;
    for(j = 2; j <= i / 2; j++) {
        if (i % j == 0) {
            return 0;
        }
    }
    return 1;
}

int main() {
    int i;
    for(i = 2; i < 100000; i++) {
        if(function(i)) {
            printf("%d\n", i);
        }
    }
}

Then compile, run, and report:

$ gcc -g -O0 test.c && perf record ./a.out && perf report

The output looks something like this:

  72.38%  a.out  a.out              [.] 0x0000000000000544
   8.37%  a.out  a.out              [.] 0x000000000000055a
   8.30%  a.out  a.out              [.] 0x000000000000053d
   7.81%  a.out  a.out              [.] 0x0000000000000551
   0.40%  a.out  a.out              [.] 0x0000000000000540

This does not have symbols, even though the executable does have symbol information.

Doing the same general steps outside the container works fine, and shows something like this:

96.96%  a.out  a.out             [.] function 
0.35%  a.out  libc-2.19.so       [.] _IO_file_xsputn@@GLIBC_2.2.5
0.14%  a.out  [kernel.kallsyms]  [k] update_curr
0.12%  a.out  [kernel.kallsyms]  [k] update_cfs_shares
0.11%  a.out  [kernel.kallsyms]  [k] _raw_spin_lock_irqsave                 

In the host system I have already turned on kernel symbols by becoming root and doing:

$ echo 0 > /proc/sys/kernel/kptr_restrict 

How do I get the containerized version to work properly and show debugging symbols?

Answer 1

Running the container with -v /:/host flag and running perf report in the container with --symfs /host flag fixes it:

 96.59%  a.out  a.out              [.] function
  2.93%  a.out  [kernel.kallsyms]  [k] 0xffffffff8105144a
  0.13%  a.out  [nvidia]           [k] 0x00000000002eda57
  0.11%  a.out  libc-2.19.so       [.] vfprintf
  0.11%  a.out  libc-2.19.so       [.] 0x0000000000049980
  0.09%  a.out  a.out              [.] main
  0.02%  a.out  libc-2.19.so       [.] _IO_file_write
  0.02%  a.out  libc-2.19.so       [.] write

Part of the reason why it doesn't work as is? The output from perf script sort of sheds some light on this:

...
           a.out    24 3374818.880960: cycles:  ffffffff81141140 __perf_event__output_id_sample ([kernel.kallsyms])
           a.out    24 3374818.881012: cycles:  ffffffff817319fd _raw_spin_lock_irqsave ([kernel.kallsyms])
           a.out    24 3374818.882217: cycles:  ffffffff8109aba3 ttwu_do_activate.constprop.75 ([kernel.kallsyms])
           a.out    24 3374818.884071: cycles:            40053d [unknown] (/var/lib/docker/aufs/diff/9bd2d4389cf7ad185405245b1f5c7d24d461bd565757880bfb4f970d3f4f7915/a.out)
           a.out    24 3374818.885329: cycles:            400544 [unknown] (/var/lib/docker/aufs/diff/9bd2d4389cf7ad185405245b1f5c7d24d461bd565757880bfb4f970d3f4f7915/a.out)
...

Note the /var/lib/docker/aufs path. That's from the host so it won't exist in the container and you need to help perf report to locate it. This likely happens because the mmap events are tracked by perf outside of any cgroup and perf does not attempt to remap the paths.

Another option is to run perf host-side, like sudo perf record -a docker run -ti <container name>. But the collection has to be system-wide here (the -a flag) as containers are spawned by docker daemon process which is not in the process hierarchy of the docker client tool we run here.

Answer 2

Another way that doesn't require changing how you run the container (so you can profile an already running process) is to mount container's root on host using bindfs:

bindfs /proc/$(docker inspect --format {{.State.Pid}} $CONTAINER_ID)/root /foo

Then run perf report as perf report --symfs /foo

You'll have to run perf record system wide, but you can restrict it to only collect events for the specific container:

perf record -g -a -F 100 -e cpu-clock -G docker/$(docker inspect --format {{.Id}} $CONTAINER_ID) sleep 90