When golang does allocation for string to byte conversion

Question

var testString = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
//var testString = "ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZ"
func BenchmarkHashing900000000(b *testing.B){
    var bufByte = bytes.Buffer{}
    for i := 0; i < b.N ; i++{
        bufByte.WriteString(testString)
        Sum32(bufByte.Bytes())
        bufByte.Reset()
    }
}

func BenchmarkHashingWithNew900000000(b *testing.B){
    for i := 0; i < b.N ; i++{
        bytStr := []byte(testString)
        Sum32(bytStr)
    }
}

test result:

With  testString = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
BenchmarkHashing900000000-4         50000000            35.2 ns/op         0 B/op          0 allocs/op
BenchmarkHashingWithNew900000000-4  50000000            30.9 ns/op         0 B/op          0 allocs/op

With testString = "ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZ"
BenchmarkHashing900000000-4         30000000            46.6 ns/op         0 B/op          0 allocs/op
BenchmarkHashingWithNew900000000-4  20000000            73.0 ns/op        64 B/op          1 allocs/op

Why there is allocation in case of BenchmarkHashingWithNew900000000 when string is long but no allocation when string is small.
Sum32 : https://gowalker.org/github.com/spaolacci/murmur3
I am using go1.6

Answer 1

Your benchmarks are observing a curious optimisation by the Golang compiler (version 1.8).

You can see the PR from Dmitry Dvyukov here

https://go-review.googlesource.com/c/go/+/3120

Unfortunately that is from a long time ago, when the compiler was written in C, I am not sure where to find the optimisation in the current compiler. But I can confirm that it still exists, and Dmitry's PR description is accurate.

If you want a clearer self contained set of benchmarks to demonstrate this I have a gist here.

https://gist.github.com/fmstephe/f0eb393c4ec41940741376ab08cbdf7e

If we look only at the second benchmark BenchmarkHashingWithNew900000000 we can see a clear spot where it 'should' allocate.

bytStr := []byte(testString)

This line must copy the contents of testString into a new []byte. However in this case the compiler can see that bytStr is never used again after Sum32 returns. Therefore it can be allocated on the stack. However, as strings can be arbitrarily large a limit is set to 32 bytes for a stack allocated string or []byte.

It's worth being aware of this little trick, because it can be easy to trick yourself into believing some code does not allocate, if your benchmark strings are all short.