Why does json.Encoder add an extra line?

Question

json.Encoder seems to behave slightly different than json.Marshal. Specifically it adds a new line at the end of the encoded value. Any idea why is that? It looks like a bug to me.

package main

import "fmt"
import "encoding/json"
import "bytes"

func main() {
    var v string
    v = "hello"
    buf := bytes.NewBuffer(nil)
    json.NewEncoder(buf).Encode(v)
    b, _ := json.Marshal(&v)

    fmt.Printf("%q, %q", buf.Bytes(), b)
}

This outputs

"\"hello\"\n", "\"hello\""

Try it in the Playground

Answer 1

Because they explicitly added a new line character when using Encoder.Encode. Here's the source code to that func, and it actually states it adds a newline character in the documentation (see comment, which is the documentation):

https://golang.org/src/encoding/json/stream.go?s=4272:4319

// Encode writes the JSON encoding of v to the stream,
// followed by a newline character.
//
// See the documentation for Marshal for details about the
// conversion of Go values to JSON.
func (enc *Encoder) Encode(v interface{}) error {
    if enc.err != nil {
        return enc.err
    }
    e := newEncodeState()
    err := e.marshal(v)
    if err != nil {
        return err
    }
    
    // Terminate each value with a newline.
    // This makes the output look a little nicer
    // when debugging, and some kind of space
    // is required if the encoded value was a number,
    // so that the reader knows there aren't more
    // digits coming.
    e.WriteByte('\n')

    if _, err = enc.w.Write(e.Bytes()); err != nil {
        enc.err = err
    }
    encodeStatePool.Put(e)
    return err
}

Now, why did the Go developers do it other than "makes the output look a little nice"? One answer:

Streaming

The go json Encoder is optimized for streaming (e.g. MB/GB/PB of json data). It is typical that when streaming you need a way to deliminate when your stream has completed. In the case of Encoder.Encode(), that is a \n newline character. Sure, you can certainly write to a buffer. But you can also write to an io.Writer which would stream the block of v.

This is opposed to the use of json.Marshal which is generally discouraged if your input is from an untrusted (and unknown limited) source (e.g. an ajax POST method to your web service - what if someone posts a 100MB json file?). And, json.Marshal would be a final complete set of json - e.g. you wouldn't expect to concatenate a few 100 Marshal entries together. You'd use Encoder.Encode() for that to build a large set and write to the buffer, stream, file, io.Writer, etc.

Whenever in doubt if it's a bug, I always lookup the source - that's one of the advantages to Go, it's source and compiler is just pure Go. Within [n]vim I use \gb to open the source definition in a browser with my .vimrc settings.