When appending to a [][]string
profiling shows the app uses around 145MiB of memory.
defer profile.Start(profile.MemProfile).Stop()
f, _ := os.Open("test.csv") // 100 MiB File
r := csv.NewReader(f)
var records [][]string
for {
values, err := r.Read()
if err == io.EOF {
break
}
records = append(records, values)
}
When storing the slice in a struct and appending that the app uses around 260MiB of memory.
defer profile.Start(profile.MemProfile).Stop()
type record struct {
values []string
}
f, _ := os.Open("test.csv") // 100 MiB File
r := csv.NewReader(f)
var records []record
for {
values, err := r.Read()
if err == io.EOF {
break
}
r := record{values: values}
records = append(records, r)
}
It feels like it's using double the memory in the second example. Can someone explain why the second example uses more memory?
For those of you who are using between go 1.12 and go 1.15, debug.FreeOSMemory()
would not have returned the free memory back to OS, so htop/top will show wrong number or if you rely on RSS to monitor your app, it would be wrong.
This is due to the fact that runtime in Golang (1.12 - 1.15) uses MADV_FREE
rather than MADV_DONTNEED
.<= Go (1.11) and Go (1.16 - which was released couple of days back) use MADV_DONTNEED
.
Go 1.16 reverted back to MADV_DONTNEED
.
Please find the changelog image and change log URL here.
Please upgrade to get predicatable analytics on memory usage. If you want to use GoLang(1.12-1.15) and still want to use MADV_DONTNEED
, Kindly run your binaries using GODEBUG=madvdontneed=1
./main.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With