Menu
I am new to Go and trying to figure out how it manages memory consumption.
I have trouble with memory in one of my test projects. I don't understand why Go uses more and more memory (never freeing it) when my program runs for a long time.
I am running the test case provided below. After the first allocation, program uses nearly 350 MB of memory (according to ActivityMonitor). Then I try to free it and ActivityMonitor shows that memory consumption doubles. Why?
I am running this code on OS X using Go 1.0.3.
What is wrong with this code? And what is the right way to manage large variables in Go programs?
I had another memory-management-related problem when implementing an algorithm that uses a lot of time and memory; after running it for some time it throws an "out of memory" exception.
package main import ("fmt" "time" ) func main() { fmt.Println("getting memory") tmp := make([]uint32, 100000000) for kk, _ := range tmp { tmp[kk] = 0 } time.Sleep(5 * time.Second) fmt.Println("returning memory") tmp = make([]uint32, 1) tmp = nil time.Sleep(5 * time.Second) fmt.Println("getting memory") tmp = make([]uint32, 100000000) for kk, _ := range tmp { tmp[kk] = 0 } time.Sleep(5 * time.Second) fmt.Println("returning memory") tmp = make([]uint32, 1) tmp = nil time.Sleep(5 * time.Second) return } 
851
Go supports automatic memory management, such as automatic memory allocation and automatic garbage collection, which avoids many lurking bugs. Memory block is a continuous memory segment and memory blocks may have different sizes. In Go, memory block may host multiple value such as struct, array and Slice etc.
Go is a language which supports automatic memory management, such as automatic memory allocation and automatic garbage collection. So Go programmers can do programming without handling the underlying verbose memory management.
As Go doesn't use malloc to get memory, but asks OS directly (via mmap ), it has to implement memory allocation and deallocation on it's own (like malloc does). Go's memory allocator is originally based off TCMalloc: Thread-Caching Malloc.
Profiling. Profiling is useful for identifying expensive or frequently called sections of code. The Go runtime provides profiling data in the format expected by the pprof visualization tool. The profiling data can be collected during testing via go test or endpoints made available from the net/http/pprof package.
Currently, go uses a mark-and-sweep garbage collector, which in general does not define when the object is thrown away.
However, if you look closely, there is a go routine called sysmon which essentially runs as long as your program does and calls the GC periodically:
// forcegcperiod is the maximum time in nanoseconds between garbage // collections. If we go this long without a garbage collection, one // is forced to run. // // This is a variable for testing purposes. It normally doesn't change. var forcegcperiod int64 = 2 * 60 * 1e9 (...) // If a heap span goes unused for 5 minutes after a garbage collection, // we hand it back to the operating system. scavengelimit := int64(5 * 60 * 1e9) forcegcperiod determines the period after which the GC is called by force. scavengelimit determines when spans are returned to the operating system. Spans are a number of memory pages which can hold several objects. They're kept for scavengelimit time and are freed if no object is on them and scavengelimit is exceeded.
Further down in the code you can see that there is a trace option. You can use this to see, whenever the scavenger thinks he needs to clean up:
$ GOGCTRACE=1 go run gc.go gc1(1): 0+0+0 ms 0 -> 0 MB 423 -> 350 (424-74) objects 0 handoff gc2(1): 0+0+0 ms 1 -> 0 MB 2664 -> 1437 (2880-1443) objects 0 handoff gc3(1): 0+0+0 ms 1 -> 0 MB 4117 -> 2213 (5712-3499) objects 0 handoff gc4(1): 0+0+0 ms 2 -> 1 MB 3128 -> 2257 (6761-4504) objects 0 handoff gc5(1): 0+0+0 ms 2 -> 0 MB 8892 -> 2531 (13734-11203) objects 0 handoff gc6(1): 0+0+0 ms 1 -> 1 MB 8715 -> 2689 (20173-17484) objects 0 handoff gc7(1): 0+0+0 ms 2 -> 1 MB 5231 -> 2406 (22878-20472) objects 0 handoff gc1(1): 0+0+0 ms 0 -> 0 MB 172 -> 137 (173-36) objects 0 handoff getting memory gc2(1): 0+0+0 ms 381 -> 381 MB 203 -> 202 (248-46) objects 0 handoff returning memory getting memory returning memory As you can see, no gc invoke is done between getting and returning. However, if you change the delay from 5 seconds to 3 minutes (more than the 2 minutes from forcegcperiod), the objects are removed by the gc:
returning memory scvg0: inuse: 1, idle: 1, sys: 3, released: 0, consumed: 3 (MB) scvg0: inuse: 381, idle: 0, sys: 382, released: 0, consumed: 382 (MB) scvg1: inuse: 1, idle: 1, sys: 3, released: 0, consumed: 3 (MB) scvg1: inuse: 381, idle: 0, sys: 382, released: 0, consumed: 382 (MB) gc9(1): 1+0+0 ms 1 -> 1 MB 4485 -> 2562 (26531-23969) objects 0 handoff gc10(1): 1+0+0 ms 1 -> 1 MB 2563 -> 2561 (26532-23971) objects 0 handoff scvg2: GC forced // forcegc (2 minutes) exceeded scvg2: inuse: 1, idle: 1, sys: 3, released: 0, consumed: 3 (MB) gc3(1): 0+0+0 ms 381 -> 381 MB 206 -> 206 (252-46) objects 0 handoff scvg2: GC forced scvg2: inuse: 381, idle: 0, sys: 382, released: 0, consumed: 382 (MB) getting memory The memory is still not freed, but the GC marked the memory region as unused. Freeing will begin when the used span is unused and older than limit. From scavenger code:
if(s->unusedsince != 0 && (now - s->unusedsince) > limit) { // ... runtime·SysUnused((void*)(s->start << PageShift), s->npages << PageShift); } This behavior may of course change over time, but I hope you now get a bit of a feel when objects are thrown away by force and when not.
As pointed out by zupa, releasing objects may not return the memory to the operating system, so on certain systems you may not see a change in memory usage. This seems to be the case for Plan 9 and Windows according to this thread on golang-nuts.
147
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
