How can I gdb attach to a process running in a docker container?

Question

I have a long-running process in a docker container to which I would like to attach gdb to look at what threads are running and get stacktraces. I can attach to the process from the host, but I cannot resolve any symbols because the executable file is in a different location in the filesystem (it's in a docker-mounted volume) and the shared system libraries are all stuck in a docker filesystem image somewhere in /var/lib/docker.

I am able to generate a core file and use gdb to look at it by specifying the host's path to the executable file, but because the system libraries are all in the wrong places and are loaded in the wrong locations in the corefile, I get no information from that.

Do I have any options that I've overlooked?

Answer 1

You can attach to the process with gdb instance running in your container by attaching to the running container via lxc-attach.

Note: gdb has to be already installed in that container or you have to install it.

# find your container ID
sudo docker ps
# list of your containers - container ID is 1234567890
# find your full container ID
sudo docker ps --no-trunc -q| grep <short ID>
sudo lxc-attach -n <container long ID>

root@1234567890:/#
# optionally, you can install gdb now if it is not installed
# yum install gdb

root@1234567890:/# gdb
...
(gdb) attach 1

UPDATE 2017-04:

There is an easier workflow using docker exec now available (thanks to @42n4).

# find your container ID
sudo docker ps
# list of your containers - container ID is 1234567890
docker exec -i -t 1234567890 /bin/bash

root@1234567890:/#
# optionally, you can install gdb now if it is not installed
# yum install gdb

root@1234567890:/# gdb
...
(gdb) attach 1