Go: memory issues

Question

I need your wisdom.

I have a huge daemon written in Go. Some time ago a user reported that there might be a memory leak somewhere in the code.

I started investigating the issue. When primary code inspection didn't lead me to any clues about the nature of this leakage, I tried to focus on how my process works.

My idea was simple: if I failed to remove references to certain objects, my heap should be constantly growing. I wrote the following procedure to monitor heap:

func PrintHeap() {
    ticker := time.NewTicker(time.Second * 5)
    for {
        <-ticker.C
        st := &runtime.MemStats{}
        runtime.ReadMemStats(st)
        // From Golang docs: HeapObjects increases as objects are allocated
        // and decreases as the heap is swept and unreachable objects are
        // freed.
        fmt.Println("Heap allocs:", st.Mallocs, "Heap frees:",
            st.Frees, "Heap objects:", st.HeapObjects)
    }
}

This procedure prints some info about heap each 5 seconds, including the number of objects currently allocated.

Now a few words about what the daemon does. It processes lines from some UDP input. Each line bears some info about a certain HTTP request and is parsed into a typical Go struct. This struct has some numeric and string fields, including one for request path. Then lots of things happens to this struct, but those things are irrelevant here.

Now, I set the input rate to 1500 lines per second, each line being rather short (you may read this as: with standard request path, /).

After running the application I could see that heap size stabilizes at some point of time:

Heap allocs: 180301314 Heap frees: 175991675 Heap objects: 4309639
Heap allocs: 180417372 Heap frees: 176071946 Heap objects: 4345426
Heap allocs: 180526254 Heap frees: 176216276 Heap objects: 4309978
Heap allocs: 182406470 Heap frees: 177496675 Heap objects: 4909795
Heap allocs: 183190214 Heap frees: 178248365 Heap objects: 4941849
Heap allocs: 183302680 Heap frees: 178958823 Heap objects: 4343857
Heap allocs: 183412388 Heap frees: 179101276 Heap objects: 4311112
Heap allocs: 183528654 Heap frees: 179181897 Heap objects: 4346757
Heap allocs: 183638282 Heap frees: 179327221 Heap objects: 4311061
Heap allocs: 185609758 Heap frees: 181330408 Heap objects: 4279350

When this state was reached, memory consumption stopped to grow.

Now, I changed my input in such a way that each line became more than 2k chars long (with a huge /AAAAA... request path), and that's where weird things started to happen.

Heap size grew drastically, but still became sort of stable after some time:

Heap allocs: 18353000513 Heap frees: 18335783660 Heap objects: 17216853
Heap allocs: 18353108590 Heap frees: 18335797883 Heap objects: 17310707
Heap allocs: 18355134995 Heap frees: 18336081878 Heap objects: 19053117
Heap allocs: 18356826170 Heap frees: 18336182205 Heap objects: 20643965
Heap allocs: 18366029630 Heap frees: 18336925394 Heap objects: 29104236
Heap allocs: 18366122614 Heap frees: 18336937295 Heap objects: 29185319
Heap allocs: 18367840866 Heap frees: 18337205638 Heap objects: 30635228
Heap allocs: 18368909002 Heap frees: 18337309215 Heap objects: 31599787
Heap allocs: 18369628204 Heap frees: 18337362196 Heap objects: 32266008
Heap allocs: 18373482440 Heap frees: 18358282964 Heap objects: 15199476
Heap allocs: 18374488754 Heap frees: 18358330954 Heap objects: 16157800

But memory consumption grew liearly and never stopped. My question is: any ideas about what's going on?

I thought about memory fragmentation due to lots of huge objects, but actually I don't really know what to think.

Answer 1

You could try the go memory profiling tools.

First you need to alter your program so it provides a memory profile. There are several ways to this.

1. You can use package net/http/pprof see https://golang.org/pkg/net/http/pprof/ if it is ok for you to publish that endpoint.
2. You can use package runtime/pprof and make your program dump a memory profile to a known location in response to specific events like receiving a signal or something.

After that you can analyse the memory profile using the go tool pprof which you can either invoke as go tool pprof <path/to/executable> <file> if you chose to dump a memory profile to a file or as go tool pprof <path/to/executable> http://<host>:<port>/debug/pprof/heap if you used net/http/pprof and use top5 to get the top 5 functions, which allocated most of your memory. You can than use the list command for specific functions to see which lines allocated how much memory.

Starting from that, you should be able to reason about the increase in memory you are observing.

You can also read about this at https://blog.golang.org/profiling-go-programs which also describes how to profile your cpu usage. Just search for the word memprofile to jump to the relevant parts.