strace to monitor Dockerized application activity

Question

My goal is to monitor which ports are opened and closed by a multi-process application. My plan is to run the application in a Docker container, in order to isolate it, and then use strace to report the application activity.

I've tried with Apache server dockerized :

strace -f -o /tmp/docker.out docker run -D -P apache

I don't see any line in the report file that shows that the application accept a connection in a socket.

Can strace report the activity of processes inside the container?

Answer 1

we can add param --security-opt=seccomp:unconfined

I have try it ,it work well!!

docker run -it   --security-opt=seccomp:unconfined  centos:7 /bin/bash

yum install strace
strace ls

execve("/usr/bin/ls", ["ls"], [/* 8 vars */]) = 0
brk(NULL)                               = 0x1d0a000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 
0) = 0x7ffb588da000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or 
directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3

refer:http://johntellsall.blogspot.com/2016/10/tip-use-strace-to-debug-issues-inside.html

Answer 2

I just managed to strace a docker container using these steps:

1. Work out what distro image the container is based on, then obtain the strace binary from that distro, e.g. by installing the corresponding distro package from a container created for that purpose.

2. Copy the strace binary into a volume you can mount into the container.

3. Also create a small wrapper shell script called entry.sh which contains your strace invocation. In my case I wrote it like this:

   #!/bin/sh
   exec /path/to/strace -ff -o /path/to/dumps /bin/bash /original/entrypoint
   

   This is assuming that the original entry point, which you read from the Dockerfile of the image you want to debug, started with #!/bin/bash. Make sure you set the execute bit of this script, and place it where you also placed the strace binary.

4. Launch docker using a command like this:

   docker run -v $PWD/shared:/path/to \
              --entrypoint="/path/to/entry.sh" \
              --cap-add SYS_PTRACE \
              image-name
   

The mounted volume will make the strace and entry.sh available inside the docker. The entry point will do the strace invocation before calling the actual entry point. This might potentially cause some trouble with strace itself becoming pid 1 in the container, and failing to reap children. If that's a problem, a different approach like the one Phil suggested would be better. Finally that added capability tells docker that it's OK to start tracing processes. Otherwise you'd get error messages like

strace: …: PTRACE_TRACEME doesn't work: Operation not permitted

Actually pointing out this capability setting is the reason I'm writing my answer. I had already done the steps except for this flag, and while searching for a solution there I found both this question here and a blog post by John Goulah containing that bit of information. For the sake of completeness I think the flag should be mentioned here, too. Haven't tried Phil's approach yet, so I definitely don't claim my approach to be superior to what he suggested. I guess it might work more easily on systems where you don't want to mess with the docker daemon.

Answer 3

Take a look at the solution described at https://medium.com/@rothgar/how-to-debug-a-running-docker-container-from-a-separate-container-983f11740dc6 which tells you how to fire up a container with strace installed, which is in the same pid and network namespace as the container/process you wish to run strace against.

This is nice since it means you don't need to install strace in the container you wish to debug.

The guts of it is that when debugging a container (caddy in the example below) you run a docker container called strace and with appropriate tools installed:

docker run -t --pid=container:caddy \
  --net=container:caddy \
  --cap-add sys_admin \
  --cap-add sys_ptrace \
  strace

Assuming you make it so when building your strace container, you'll now have a shell with appropriate tools, from which you can run ps and see the process in the caddy container, and you can run strace against it.

You will be in a different container, with a different file system, but you can see the file-space of the target container at /prof/$PID/root.