Datadog % Mem Usage and RSS Memory

Question

We have a .net core service running inside a docker container which is used for processing files uploaded by user. Some of the incoming requests are with files as large as 20 MB. The application saves the files to Aurora DB as binary blobs and does other processing steps. Recently the load on service increased and the datadog shows % Mem Usage as around 99%. CPU utilization is only around 2%. We expected memory leak and consequently container running out of memory. But process seems to be running fine for some time now. Additionally, the RSS memory is 15% only (650 MB on 4 GB container). I am trying to understand the difference in these two memory metrics, and is the high memory usage result of memory leak. The process is critical and we want to take all necessary steps to ensure stability. We are analyzing the application for Memory optimizations, but couldn't identify anything major so far.

Answer 1

This sounds very similar to what our team has experienced with Datadog. For some time, we were getting alerts that our pods were running out of memory. Restarting those pods would usually solve the problem. Graphs were showing a steady increase in memory usage over one month until it reached its peak.

We dug into this issue and found out that datadog was reporting used + cached memory which resulted in incorrect metrics. The OS was caching file handles. Usually, cached memory is available for use and is freed automatically by the operating system when more memory is needed. It is also possible to flush it manually as shown here.

To confirm this, you can get into the container and run cat /proc/meminfo. This will give you a breakdown of the container's memory usage. Here is what I got

MemTotal:       6504577 kB
MemFree:        1888762 kB
MemAvailable:   4357572 kB
Buffers:            310 kB
Cached:         2230122 kB

As you can see, MemAvailable is much larger than MemFree and there is a lot of cached data sitting in memory.