Prometheus (Docker): determine available memory per node (which metric is correct?)

Question

We have been struggling to create a good memory monitoring for our nodes running Docker components. We use Prometheus in combination with cadvisor and node_exporter.

What is the best way to determine the used memory per node?

Method 1: gives in our example around 42%

(1-(node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes))*100

Method 2: gives around 80%

(1-((node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes)/node_memory_MemTotal_bytes))*100

Q2: Why is this difference? What can I learn from this?

So I digged a bit deeper on determined the individual metrics:

1. Free memory: in our experiment was about 5%

   (node_memory_MemFree_bytes/node_memory_MemTotal_bytes)*100

2. Buffered memory: around 0.002%

   (node_memory_Buffers_bytes/node_memory_MemTotal_bytes)*100

3. Cached memory: around 15%

   (node_memory_Cached_bytes/node_memory_MemTotal_bytes)*100

4. Availabable memory: 58%

   (node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes)*100

I would expect that FreeMem + BufferedMem + CachedMem would be around the AvailableMem. But that is not the outcome of this simple experiment.

Q3: Why is this not true?

It is said that the free memory on Linux consists of free mem + buffered mem + cached mem. When there is a short of memory, the cached memory could be freed, etc.

Answer 1

This documentation tells in detail what are those numbers mean: https://github.com/torvalds/linux/blob/master/Documentation/filesystems/proc.rst#meminfo

MemAvailable: An estimate of how much memory is available for starting new applications, without swapping. Calculated from MemFree, SReclaimable, the size of the file LRU lists, and the low watermarks in each zone. The estimate takes into account that the system needs some page cache to function well, and that not all reclaimable slab will be reclaimable, due to items being in use. The impact of those factors will vary from system to system.

So the MemAvailable is an estimation how much memory can be used without swapping for new processes. FreeMem is only a part that is calculated into MemAvailable. BufferedMem and CachedMem might be taken into the estimation, but these are just a smaller section of the memory that might be reclaimed:

Buffers: Relatively temporary storage for raw disk blocks shouldn't get tremendously large (20MB or so)

Cached: in-memory cache for files read from the disk (the pagecache). Doesn't include SwapCached