Trying to locate a leak! What does anon mean for pmap?

Question

I'm trying to locate where my memory has gone for a java process running in linux. Someone suggested I use pmap -x to see exactly what the memory is doing.

The output is really long but basically a good portion of it is a repeat of this:

00007fbf75f6a000    1016       -       -       - rwx--    [ anon ]
00007fbf76068000      12       -       -       - -----    [ anon ]

What exactly does this mean? Why do I have so many entries of this (4000+)?

Answer 1

Anon blocks are "large" blocks allocated via malloc or mmap -- see the manpages. As such, they have nothing to do with the Java heap (other than the fact that the entire heap should be stored in just such a block).

In my experience, thread stacks also use anon blocks. If you see a lot of anon blocks that all have the same size, and that size is 512k to 4Mb (the example below is repeated over a dozen times for a Tomcat process that I have running), that's the likely cause. Depending on the program, you may have up to a few dozen of these; if you're seeing thousands, it means you have a problem with threading.

b089f000    504K rwx--    [ anon ]
b091d000     12K -----    [ anon ]
b0920000    504K rwx--    [ anon ]
b099e000     12K -----    [ anon ]
b09a1000    504K rwx--    [ anon ]
b0a1f000     12K -----    [ anon ]

But that leaves a question: why are you using pmap to diagnose a Java memory issue?

Answer 2

See this part this part of System Performance Tuning for anonymous memory.

Answer 3

I've seen that pattern before in a thread leak. If you have code that is trying to pool threads, but somehow messes up and leaks a thread, you get a pattern like that in pmap.

I think each bit of memory is the minimum stack size for the thread, certainly it had nothing to do with heap in our case.
We still got OutOfMemoryErrors when we hit OS limits, even tho when we analise the heap it is not overallocated.

When we had a problem like this pmap [pid] | grep -c 12K turned out to be the number of threads in use.