How to record the peak memory usage of a Docker container?

Question

I'm running a number of short-lived docker containers each of which does some memory-intensive batch processing. I'm looking for a way to find the peak memory usage each container hit while it was running. Knowing this will allow me to optimize the infrastructure I run these containers on for future runs.

One naive way to achieve this is redirecting the streaming output of docker stats to some file: docker stats container_id > stats.log. However, this requires running a process for each container and then sorting through very verbose logs to find the peak usage. I'm wondering if there's not an easier way.

Answer 1

If you are interested in the process with PID=1 inside the container, you can find the PID this process has on the host and then use:

grep VmPeak /proc/$PID/status

---

Example with a mongo container:

This container has a single process:

$ docker container exec -it mongo top -bn 1
top - 10:04:51 up 32 min,  0 users,  load average: 0.36, 0.52, 0.55
Tasks:   2 total,   1 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.1 us,  2.0 sy,  0.3 ni, 90.6 id,  0.7 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :  6103572 total,  2642744 free,  1352032 used,  2108796 buff/cache
KiB Swap:  1942896 total,  1942896 free,        0 used.  4277928 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   64 root      20   0   38624   3116   2724 R   6.7  0.1   0:00.89 top
    1 mongodb   20   0 1094540  80100  35916 S   0.0  1.3   0:22.51 mongod

To get the PID of this process from the host's perspective:

$ docker inspect -f '{{.State.Pid}}' mongo
2532

and finally:

$ grep VmPeak /proc/2532/status
VmPeak:  1094540 kB

---

Links:

• Peak memory usage of a linux/unix process
• Peak memory usage of a process
• Finding Docker container processes? (from host point of view)