Difference in json.Unmarshal when using bytes.Buffer vs using *bytes.NewBuffer

Question

I was looking at the bytes package. If I define a buffer using bytes.Buffer then the code below works and i get an output. However if I try create a buffer with a certain capacity and then try the same code it fails with the error error: invalid character '\x00' looking for beginning of value. Not sure how to fix it.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
)

func main() {
    var jsonBlob = []byte(`[
        {"Name": "Platypus", "Order": "Monotremata"},
            {"Name": "Quoll",    "Order": "Dasyuromorphia"}
            ]`)

    //var b bytes.Buffer
    b := *bytes.NewBuffer(make([]byte, 20))
    b.Write(jsonBlob)

    fmt.Println(b.String())

    var dat interface{}
    err := json.Unmarshal(b.Bytes(), &dat)
    if err != nil {
        fmt.Println("error:", err)
    }   
    fmt.Printf("%+v", dat)
}

Output for Run with bytes.Buffer

[
        {"Name": "Platypus", "Order": "Monotremata"},
        {"Name": "Quoll",    "Order": "Dasyuromorphia"}
    ]
[map[Name:Platypus Order:Monotremata] map[Name:Quoll Order:Dasyuromorphia]]
Program exited.

Output for Run with bytes.NewBuffer

[
        {"Name": "Platypus", "Order": "Monotremata"},
        {"Name": "Quoll",    "Order": "Dasyuromorphia"}
    ]
error: invalid character '\x00' looking for beginning of value
<nil>

Answer 1

The NewBuffer function uses the argument as the initial contents of the buffer. The call make([]byte, 20) returns a byte slice containing 20 zero bytes. The contents of the buffer after b.Write(jsonBlob) is 20 zero bytes followed by the JSON text.

Add fmt.Printf("%q\n", b.String()) to the program to view the contents of the buffer.

playground example with printf added

The JSON parser complains about the first zero byte.

If your goal is to set the size of the internal buffer, use this code:

b := bytes.NewBuffer(make([]byte, 0, 20))

The call make([]byte, 0, 20) returns a zero length slice with capacity 20.

playground example with zero length slice

A variable of type byte.Buffer starts as the empty buffer.

If your goal is to limit the amount of data read, then use io.LimitedReader. For example:

f, err := os.Open("filename")
if err != nil {
   // handle error
}
defer f.Close()
err := json.NewDecoder(&io.LimitedReader{N: 20, R: f}).Decode(&dat)
if err != nil {
    // handle error. Will get parse error if file is truncated. 
}

Answer 2

read this:

func NewBuffer

func NewBuffer(buf []byte) *Buffer NewBuffer creates and initializes a new Buffer using buf as its initial contents. It is intended to prepare a Buffer to read existing data. It can also be used to size the internal buffer for writing. To do that, buf should have the desired capacity but a length of zero.

In most cases, new(Buffer) (or just declaring a Buffer variable) is sufficient to initialize a Buffer.

After b.Write(jsonBlob), because your buffer din't have a length of zero(make([]byte, 20) creates a 20-length slice), so that b.Bytes() is 20 bytes you allocated plus the json content. Then when you do Unmarshal, the json parser will see 20 zeros in the beginning, of course it complains.