Where has my used memory gone?

Question

I got a linux hardware server having 16GB of physical memory and running some applications. This server is up and running for around 365 days till now and I am observing the "free -m" showing memory is running low.

             total       used       free     shared    buffers     cached
Mem:         14966      13451       1515          0        234        237
-/+ buffers/cache:      12979       1987
Swap:         4094        367       3727

I understand 1987 is the actual free memory in the system which less than 14%. If I add up the %MEM section in "ps -A v" output or from "top" it does not add up to 100%.

I need to understand why the memory has gone so low?

Update (29/Feb/2012):
Let me split this problem into two parts:
1) System having less free memory.
2) Identifying where the used memory has gone.

For 1), I understand; if system is running low on free memory we may see gradual degradation in performance. At some point paging would give additional free memory to the system resulting in restoration in system's performance. Correct me if I am wrong on this.

For 2), Now this is what I want to understand where has the used memory vanished. If I sum up the %MEM in output of "ps -A v" or "top -n 1 -b" it comes to no more than 50%. So where to account for the remaining 40% of untraceable memory. We have our own kernel modules in the server. If these modules leak memory would they get accounted. Is it possible to know amount of leakage in kernel modules.

Answer 1

It's not running low. Free memory is running low. But that's fine, since free memory is completely useless. (Free memory is memory that is providing no benefit. Free memory is memory that would be just a useful sitting on your shelf as in your computer.)

Free memory is bad, it serves no purpose. Low free memory is good, it means your system has found some use for most of your memory.

So what's bad? If your system is slow because it doesn't have enough memory in use.

Answer 2

I was able to identify and solve my issue. But it was not without the help of the information present at http://linux-mm.org/Low_On_Memory.

The memory at slabinfo for dentry was around 5GB. After issuing "sync" command the dirty pages got synced to hard-drive and the command "echo 3 > /proc/sys/vm/drop_caches" freed up some more memory by dropping some more caches.

In addition to the literature present in the above website, the memory is reclaimed by the kernel at a rate dependent on vfs_cache_pressure (/proc/sys/vm/vfs_cache_pressure).

Thanks to all for your help.