linux high slab cache usage

Question

On one of my server, I have some memory/disk KV service, Memory KV behave like memcached, ask for a big trunk of memory(10GB) when initialized, Disk Kv behave like leveldbd, its random read and sequential write, and it frequently reads a lot files. Memory are all alloced using libc malloc.

My KV server process do not consume a lot of memory as below (since lack of memory, I have killed memory KV, leaving only disk KV, but free memory still goes down):

:~$top
PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20 0 5030m 3.9g 2772 S 8 6.1 10430:52 tair_server
20 0 4833m 3.9g 4560 S 8 6.1 10171:07 tair_server
20 0 4844m 3.9g 3844 S 38 6.1 10073:32 tair_server
20 0 4765m 3.8g 4144 S 8 6.0 10552:39 tair_server
20 0 2941m 2.4g 9.8m S 0 3.8 256:45.70 tair_server
20 0 2953m 2.4g 12m S 1 3.7 276:54.64 tair_server

But, my memory are gone.

$free -m

             total       used       free     shared    buffers     cached
Mem:         64552      57778       6774          0         16        326
-/+ buffers/cache:      57435       7117
Swap:            0          0          0

I can see slab consume lots of my memory, and it's unreclaimable.

$cat /proc/meminfo
MemTotal:       66101892 kB
MemFree:         6816228 kB
Buffers:           17024 kB
Cached:           456640 kB
SwapCached:            0 kB
Active:         19697712 kB
Inactive:        3197312 kB
Active(anon):   19546504 kB
Inactive(anon):  2875632 kB
Active(file):     151208 kB
Inactive(file):   321680 kB
Unevictable:          48 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:              6612 kB
Writeback:            72 kB
AnonPages:      22421152 kB
Mapped:            54408 kB
Shmem:               332 kB
Slab:           28870400 kB
SReclaimable:     213344 kB
SUnreclaim:     28657056 kB
KernelStack:       30000 kB
PageTables:        62776 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    33050944 kB
Committed_AS:   37517224 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      388624 kB
VmallocChunk:   34324313700 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        5696 kB
DirectMap2M:     2082816 kB
DirectMap1G:    65011712 kB

Here is the slab info.

$slabtop -s c

OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME  
69842766 69838389  38%    0.19K 1663025       42  13304200K kmalloc-192
69314912 69314796  38%    0.12K 2166091       32   8664364K kmalloc-128
70866624 70866323  39%    0.06K 1107291       64   4429164K kmalloc-64
69299968 69299592  38%    0.03K 541406      128   2165624K kmalloc-32
128388  72434  56%    0.99K   4230       32    135360K ext4_inode_cache
208782  94112  45%    0.19K   4971       42     39768K dentry

I don't understand what consumes lots memory, why it does so, and how to solve this.

Can this be an interval kernel error?

OR it's a glibc problem, it do not return memory back to system, due to frequently disk read?

Answer 1

It looks like your distro version is a bit old, but that's ok. Don't listen to people who tell you that you have to upgrade before viewing your unmae -a output. However it would be nice if you supply it...

In the newer versions of server and desktop distros, the free command output and the /proc/meminfo contain one more line for the purpose of eliminating exactly that kinds of confusions that you have. The line name is "MemAvailable" in /proc/meminfo and "available" in free output.

The column "free" in free -m does not show free memory in the way that humans understand that (and so the "MemFree" line in /proc/meminfo). It does not exclude kernel's page cache and other caches which are not "used" in the way that humans understand that.

That's the first thing. If you think that I'm wrong and you understand the free output correctly try: echo 3 > /proc/sys/vm/drop_caches and see what happens then with memory usage. Please provide the output of free after executing that command as root.

If it's still that bad please read that: https://www.linuxquestions.org/questions/linux-server-73/very-high-slab-usage-hard-to-understand-901323/. It says that your kernel may have to be upgraded.