Linux memory reporting discrepancy [closed]

Question

I'm getting a memory usage discrepancy between meminfo and ps. Free is reporting much less free memory than what processes are apparently using according to ps.

According to free, I have only 3188mb free:

free -m
             total       used       free     shared    buffers     cached
Mem:         15360      13273       2086          0         79       1022
-/+ buffers/cache:      12171       3188
Swap:            0          0          0

I try to track down where the memory is going using ps (snipped below non 0 RSS values):

ps -A --sort -rss -o comm,pmem,rss

COMMAND         %MEM   RSS
mysqld          13.1 2062272
java             6.2 978072
ruby             0.7 114248
ruby             0.7 114144
squid            0.1 30716
ruby             0.0 11868
apache2          0.0 10132
apache2          0.0  9092
apache2          0.0  8504
PassengerHelper  0.0  5784
sshd             0.0  3008
apache2          0.0  2420
apache2          0.0  2228
bash             0.0  2120
sshd             0.0  1708
rsyslogd         0.0  1164
PassengerLoggin  0.0   880
ps               0.0   844
dbus-daemon      0.0   736
sshd             0.0   736
ntpd             0.0   664
squid            0.0   584
cron             0.0   532
ntpd             0.0   512
exim4            0.0   504
nrpe             0.0   496
PassengerWatchd  0.0   416
dhclient3        0.0   344
mysqld_safe      0.0   316
unlinkd          0.0   284
logger           0.0   252
init             0.0   200
getty            0.0   120

However, this doesn't make sense as adding up the RSS column leads to a total memory usage of only around 3287mb that should leave almost 12gb free!

I'm using kernel 2.6.16.33-xenU #2 SMP x86_64 on Amazon AWS.

Where is my memory going? Can anyone shed some light on how to track this down?

Answer 1

Check the usage of the Slab cache (Slab:, SReclaimable: and SUnreclaim: in /proc/meminfo). This is a cache of in-kernel data structures, and is separate from the page cache reported by free.

If the slab cache is resposible for a large portion of your "missing memory", check /proc/slabinfo to see where it's gone. If it's dentries or inodes, you can use sync ; echo 2 > /proc/sys/vm/drop_caches to get rid of them.

You can also use the slabtop tool to show the current usage of the Slab cache in a friendly format. c will sort the list by current cache size.

Answer 2

You cannot just add up the RSS or VSZ columns to get the amount of memory used. Unfortunately, memory usage on Linux is much more complicated than that. For a more thorough description see Understanding memory usage on Linux, which explains how shared libraries are shared between processes, but double-counted by tools like ps.

I don't know offhand how free computes the numbers it displays but if you need more details you can always dig up its source code.

Answer 3

I believe that you are missing the shared memory values. I don't think ps reports the shared RAM as part of the RSS field. Compare with the top RES field to see.

Of course if you do add in the shared RAM, how much do you add? Because it is shared the same RAM may show up credited to many different processes.

You can try to solve that problem by creative parsing of the /proc/[pid]/smaps files.

But still, that only gets you part of the way. Some memory pages are shared but accounted as resident. These pages get shared after a fork() call. They can become unshared at any time but until they are they don't count toward total used system RAM. The proc smaps file doesn't show these either.