Why 'Total MB' in golang heap profile is less than 'RES' in top?

Question

I have a service written in go that takes 6-7G memory at runtime (RES in top). So I used the pprof tool trying to figure out where the problem is.

go tool pprof --pdf http://<service>/debug/pprof/heap > heap_prof.pdf

But there are only about 1-2G memory in result ('Total MB' in pdf). Where's the rest ?

And I've tried profile my service with GOGC=off, as a result the 'Total MB' is exactly the same as 'RES' in top. It seems that memory is GCed but haven't been return to kernel won't be profiled.

Any idea?

P.S, I've tested in both 1.0.3 and 1.1rc3.

Answer 1

This is because Go currently does not give memory of GC-ed objects back to the operating system, to be precise, only for objects smaller then predefined limit (32KB). Instead memory is cached to speed up future allocations Go:malloc. Also, it seems that this is going to be fixed in the future TODO.

Edit: New GC behavior: If the memory is not used for a while (about 5 min), runtime will advise the kernel to remove the physical mappings from the unused virtual ranges. This process can be forced by calling runtime.FreeOSMemory()