How to know whether enough memory is free to deploy a new application on a Linux machine?

Question

I have got a Linux machine whose memory snapshot (according to /proc/meminfo) is as follows:

MemTotal:     16413388 kB MemFree:         48296 kB Buffers:        193600 kB Cached:        1986448 kB SwapCached:     874512 kB Active:       15034264 kB Inactive:       713672 kB HighTotal:           0 kB HighFree:            0 kB LowTotal:     16413388 kB LowFree:         48296 kB SwapTotal:     8385920 kB SwapFree:      4682408 kB Dirty:            3124 kB Writeback:           0 kB Mapped:       13005560 kB Slab:           257784 kB CommitLimit:  16592612 kB Committed_AS: 59624324 kB PageTables:     233748 kB VmallocTotal: 536870911 kB VmallocUsed:    267064 kB VmallocChunk: 536603555 kB HugePages_Total:     0 HugePages_Free:      0 Hugepagesize:     2048 kB 

This is a 16 GB-machine, and I have a Java application to deploy on it, which will use 3 JVM instances whose typical combined memory requirement will be close to 1 GB.

How can I make sure that it will be safe to deploy the said application without affecting other applications currently running on that machine. Is it possible to find that out from the above memory snapshot?

What other statistics may help me to decide that, and how can I collect those statistics?

Answer 1

(It's probably a bit late for the OP, but this is asked quite often, so I'll give it a shot)

free normally shows something like this:

             total       used       free     shared    buffers     cached Mem:       8195284    8137708      57576          0    1232328    2651156 -/+ buffers/cache:    4254224    3941060 Swap:     18892216     759852   18132364 

People tend to look at the Mem: line when trying to find out how much free memory they have. Unfortunately that line is quite misleading, because the Linux kernel tries to make optimal use of the available memory in (at least) these ways:

• It will cache data from the I/O subsystem (e.g. the disk), so that it will be readily available if needed.

• It will actively evict processes that have been inactive for some time to the swap space, in favour of caching data for active processes. This tends to favour throughput over responsiveness, so some people tune their kernel to change this behaviour.

The first point is the source of confusion regarding free, because the Mem: line includes the memory used for caching in the used memory amount. The kernel, though, will cache as much as possible for performance reasons. In fact, on any Linux system that has been up for some time, the free memory tends to be close to zero - unused memory is wasted memory.

The cache memory, though, can be freed by the kernel if needed by another process. While it will impact I/O performance to a degree, other processes can have more memory without using the swap space. Therefore, for most intents and purposes, that memory is free.

That's why free includes a second line, where the cache memory is considered free:

-/+ buffers/cache:    4254224    3941060 

This second line is what people should be looking at when they want to know if they have enough free memory for a certain purpose.

In the example above, according to the Mem: line there are ~57 MB of free memory. If one reads the second line, though, there are in fact about 3.9 GB that can be used without forcing active processes to swap. As a sidenote, there are also about 760 MB of rarely-used data that have been swapped out, to make more space in the main memory for processes and caching.

At roughly the same time, the contents of /proc/meminfo:

MemTotal:        8195284 kB MemFree:           57660 kB Buffers:         1232352 kB Cached:          2651156 kB SwapCached:       119936 kB . . . 

MemTotal: the available physical memory detected by the kernel.

MemFree: the unused physical memory - the free memory shown in the Mem: line of free.

Buffers: relatively temporary storage of raw disk blocks.

Cached: in-memory cache for files read from the disk. It does not include SwapCached memory.

SwapCached: memory that was once swapped out, then swapped back in but is still in the swap space. If needed, its contents can be just discarded (very fast!), without having to swap them out (slower).

So, to have a semi-accurate estimate of the memory that is actually available

MemFree + Buffers + Cached + SwapCached 

is a good starting point - and the one free shows in that second line.

Naturally, memory management and the related statistics and measurements are more complicated than this. The numbers shown by free are mere estimates at best, since there are a lot of other variables to take into account if you want to go deeper. For people who regularly perform memory usage optimization, this is almost a form of art.

EDIT:

A somewhat humorous link about this "issue":

http://www.linuxatemyram.com/

EDIT 2:

To confirm the comment about memory use analysis almost being a form of art:

Even free misses a major chunk of cached data on modern Linux systems. From /proc/meminfo on my system:

SReclaimable:    2253576 kB 

That's about 2GB of memory that is used by the system slab allocator for caching directory entries and such and it is reclaimable (i.e. it can be cleared and used by processes if necessary). Yet free does not consider it cache memory and does not enter it in any of its calculations and therefore it shows up as used memory.

The slabtop utility, if available, allows the system administrator to find out what the slab cache is used for.

A way (for the root user only) to have free show the actual memory use of the system is the following:

# swapoff -a # sync # echo 3 > /proc/sys/vm/drop_caches  # free              total       used       free     shared    buffers     cached Mem:       8195284    3181468    5013816          0       8656     228832 -/+ buffers/cache:    2943980    5251304 Swap:            0          0          0 # swapon -a 

The first command disables the swap space. It should not be issued if the available memory may not be enough to hold the data that have been swapped out - in that case one has to take into account the Swap: line of free in their memory usage calculations.

The second command pushes all buffered data to the disk. It allows more cache memory to be freed in the next step.

The third command is the most important of the set - it forces the kernel to discard as much cached data as possible (page cache, directory entries, inodes etc).

Then free finally shows what the running processes actually use in its -/+ buffers/cache: line. It is quite noticeable that even after dropping all cached data the kernel quickly starts caching again - in this case it has already reached almost 250MB of cached data within a few seconds.

The final command enables the swap space again - it is only necessary if the first command was used too.

It should be noted that these commands should be executed by the root user in order to have the necessary privileges.