Increase Heap size in GO

Question

Is there a way howto instruct GO runtime to use larger heaps? I am running GO 1.5.

My GO process is currently spending 34% of time in GC but it uses unly 1/3 of available systems memory.

I know ulimit can be used to limit max heap size. I have set ulimit to ~16GB (ulimit -v 17179869184) but the heap size never goes over 5GB.

Using GODEBUG=gctrace=1 I can see high GC overhead (34%):

20160719-220359.169294 :: gc 665 @5484.983s 34%: 3.3+2504+188+1635+8.0 ms clock, 26+2504+0+26950/3271/3.5+64 ms cpu, 4825->4964->2623 MB, 4948 MB goal, 8 P
20160719-220406.322354 :: gc 666 @5492.411s 34%: 2.9+212+2111+1749+8.3 ms clock, 23+212+0+25010/3496/146+67 ms cpu, 4846->4990->2657 MB, 4970 MB goal, 8 P
20160719-220413.703514 :: gc 667 @5499.452s 34%: 4.4+4411+0.021+0.25+8.4 ms clock, 35+4411+0+29365/0.054/38+67 ms cpu, 4908->5022->2618 MB, 5025 MB goal, 8 P

Answer 1

You can control this with the GOGC environment variable. It is a percentage: set it to 200 and the Go runtime will use twice as much memory as before.

[this was buried in the comments; I'm making it visible as an answer]

Update: there is a detailed discussion of different techniques at https://github.com/golang/go/issues/23044, including mention of the "ballast" technique.

Answer 2

The Golang documentation describes GOGC:

The GOGC variable sets the initial garbage collection target percentage. A collection is triggered when the ratio of freshly allocated data to live data remaining after the previous collection reaches this percentage. The default is GOGC=100. Setting GOGC=off disables the garbage collector entirely.

There is an optimal value for GOGC, and it depends on the application and system that you run the Go application on. You can read more in this blog and this one.

You can try different values and see the effects on performance, or use a free tool like Optimizer Studio that will find the optimal GOGC value automatically.