Docker stats 100% memory

Question

I've a container which is running a java application with the following jvm arguments:

-XX:+UseG1GC -Xms512m -Xmx2048m -XX:MaxPermSize=256m

I'm using docker memory limit option:

docker run -it -m 2304m foo bash

Running docker stats myApp right after the container initialization will give me:

CONTAINER   CPU %    MEM USAGE/LIMIT     MEM %   NET I/O
myApp       0.17%  660.5 MB/2.416 GB    27.34%   240.8 kB/133.4 kB

But after a few hours I've the following stats:

CONTAINER   CPU %    MEM USAGE/LIMIT     MEM %   NET I/O
myApp     202.18%  2.416 GB/2.416 GB   100.00%   27.67 GB/19.49 GB

Although, If I look into the process execution details of the running application inside the container, I have an usage of ~735MB and myApp continues to compute requests without any problems:

me@docker-container ~]$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
me+          1  0.0  0.0  11636  1324 ?        Ss   13:44   0:00 /bin/bash /home/bar/service/start-myApp.sh
me+          6  113  4.5 5014152 735736 ?      Sl   13:44 438:46 java -XX:+UseG1GC -Xms512m -Xmx2048m -XX:MaxPermSize=256m -jar myApp-service-1.0-final.jar
me+        481  0.0  0.0  11768  1820 ?        Ss   20:09   0:00 bash
me+        497  0.0  0.0  35888  1464 ?        R+   20:10   0:00 ps aux

Worthy to mention that I've used jconsole to monitor process 6 and everything looks good.

Why is docker container using all the memory available if its content does not need it? I expected that docker would use a little more memory than myApp... not 100% of the available memory.

Answer 1

Lets start with this:

 -XX:+UseG1GC -Xms512m -Xmx2048m -XX:MaxPermSize=256m

That says, use a heap that starts at 0.5Gb and can grow to 2GB, and also a permgen heap of 0.25GB. And that does not include the JVM's other non-heap usage; e.g. memory mapped files, thread stacks, cached JAR files, etc.

Then you say that docker is reporting that the container is using 2.416 GB. That is not surprising. 2.42 - 2.25 is 0.17GB, and that is not excessive for non-heap memory usage.

Finally, the 735736 RSS value is telling you the resident set size; i.e. the current amount of physical RAM that that process is using. The JVM arguments and the docker stats command are measures of virtual memory size.

---

Why is docker container using all the memory available if its content does not need it? I expected that docker would use a little more memory than myApp... not 100% of the available memory.

I think that you are misreading the ps aux output. The RSS is just the physical memory being used. In fact, the total memory usage of your process is given by the VSZ ... which is 5GB. Now that >does< look large, and it is not obvious why its is that large. But taking it on face value, that implies that Docker is under-reporting the containers true memory / virtual memory usage.

The other thing is that a Docker container does not isolate an application in the container from resource demands by other things outside of the container. The JVM will be competing for physical RAM with other applications inside and outside of the container.

For more information:

• https://goldmann.pl/blog/2014/09/11/resource-management-in-docker/ explains how Docker resource management works, and what it can and cannot do.

• What is RSS and VSZ in Linux memory management.