What exactly is buffer (last parameter) in io.Copybuffer(...)?

Question

I understand it's handy in reusing a buffer rather than allocating every time hen using io.Copy. However, having printed its value several times, I get all zeros and the size of my buffer never changes. I tried to set the size to 8 and 1.

On a related note, to what value should I set my buffer size?

Answer 1

io.CopyBuffer() documents that:

func CopyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error)

CopyBuffer is identical to Copy except that it stages through the provided buffer (if one is required) rather than allocating a temporary one. If buf is nil, one is allocated; otherwise if it has zero length, CopyBuffer panics.

If either src implements WriterTo or dst implements ReaderFrom, buf will not be used to perform the copy.

So io.CopyBuffer() copies data (bytes) from src to dst. The source is an io.Reader and the destination is an io.Writer. These interfaces allow you to read and write slices of bytes ([]byte).

In the general case to do the copying, we need a slice to read into from the source, which we can write into the destination. So io.CopyBuffer() needs a buffer. The buf param allows you to pass a byte slice if you already have one, and if you do so, that buffer will be used to do the job, so no new slice have to be allocated (which would be thrown away at the end of the operation).

What size should it be? The bigger the better, but no bigger is needed than the data you want to copy. Obviously bigger requires more memory, so there's a trade-off. Typically a few KB is a good compromise.

Note that as documented, if the source implements io.WriterTo or the destination implements io.ReaderFrom, those interfaces allow to read / write without having to pass a slice, so in that case the buffer you pass will not be used. Like in this example:

srcData := []byte{1, 2, 3, 4, 5, 6, 7}
src := bytes.NewBuffer(srcData)
dst := &bytes.Buffer{}
buf := make([]byte, 10)

io.CopyBuffer(dst, src, buf)

fmt.Println(srcData)
fmt.Println(dst.Bytes())
fmt.Println(buf)

Which outputs (try it on the Go Playground):

[1 2 3 4 5 6 7]
[1 2 3 4 5 6 7]
[0 0 0 0 0 0 0 0 0 0]

Since we used bytes.Buffer as the source and destination (and since it implements both io.ReaderFrom and io.WriterTo), the buffer is not used.

Let's construct a source and destination that does not implement these interfaces, so we can test if / how our passed buffer is used.

For this, I will embed *bytes.Buffer in a struct, but specify a WriteTo and ReadFrom fields, so those methods will not get promoted from the embedded bytes.Buffer:

srcData := []byte{1, 2, 3, 4, 5, 6, 7}
src := struct {
    WriteTo int // "disable" WriteTo method
    *bytes.Buffer
}{0, bytes.NewBuffer(srcData)}

dst := struct {
    ReadFrom int // "disable" ReadFrom method
    *bytes.Buffer
}{0, &bytes.Buffer{}}

buf := make([]byte, 10)

io.CopyBuffer(dst, src, buf)

fmt.Println(srcData)
fmt.Println(dst.Bytes())
fmt.Println(buf)

This will output (try it on the Go Playground):

[1 2 3 4 5 6 7]
[1 2 3 4 5 6 7]
[1 2 3 4 5 6 7 0 0 0]

As you can see, the data from the source was read into the buffer, which then was written to the destination.

Note that you may pass a buffer smaller than the data to be copied, in which case reading / writing will be done in several iterations. In such cases, the data in the buffer may hold only the last iteration, and may only hold partial data (if the copied size is not an integer multiplication of the buffer size). It also depends on how the Read() method is implemented on the source, as Read() is not required to read the full slice passed to it.

Also note that io.CopyBuffer() does not document that the data written to the passed buffer is retained, it may get cleared / zeroed. Although this clearing is not implemented for performance reasons, but you should not count on it holding valid data after io.CopyBuffer() returns.