how to find, "invalid character ',' looking for beginning of value" error message

Question

I have a short Go program that runs the go list -json command for several packages, stores the output of each run of the command in a json.RawMessage, appends each json.RawMessage into a slice of json.RawMessages, and then returns the result to the server after concatenating each of the json.RawMessages together and compacting the json. However, there is an error message that gets produced when I run json.Compact that I can't locate the source of. Googling this error message reveals that most people who seem to encounter it--whether it's for an invalid , or some other character--have a hard time finding the source of it.

invalid character ',' looking for beginning of value

The code with comments is available to view here on play.golang.org (although it won't run there) and also below.

Question: can you explain the source of this error and how to prevent it?

(Note, some of the packages were included just for testing purposes)

package main

import (
    "expvar"

    "encoding/json"

    "bytes"
    "fmt"
    "github.com/go-martini/martini"
    "github.com/zenazn/goji"
    "github.com/zenazn/goji/web"
    "go/build"
    "log"
    "math/rand"
    "net/http"
    _ "net/http/pprof"
    "os/exec"
)

type myType struct {
    J []json.RawMessage
}

var pack map[string]string

type GoList struct {
    Imports []string
}

type Import struct {
    Dir        string
    ImportPath string
    Name       string
    Target     string
    Standard   bool
    Root       string
    GoFiles    []string
    Imports    []string
    Deps       []string
}

const contentTypeJSON = "application/json"

func main() {

    http.HandleFunc("/importgraph", func(w http.ResponseWriter, r *http.Request) { importGraph(w, r) })
    http.HandleFunc("/", handler)
    http.ListenAndServe(":8080", nil)

}

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Println("Inside handler")
    fmt.Fprintf(w, "Hello world from my Go program!")
}

func importGraph(w http.ResponseWriter, r *http.Request) {

    pack = make(map[string]string)

    var t myType
    cmd := exec.Command("go", "list", "-json")
    stdout, err := cmd.Output()
    if err != nil {

        println(err.Error())
        return
    }

    var list GoList
    err = json.Unmarshal(stdout, &list)

    for _, d := range list.Imports {
        //get the imports for each of the packages listed by go list -json
        t.imports(d)

    }

    var buff bytes.Buffer

    //concatenate the separate json.RawMessages together into json

    buff.WriteByte('[')

    for i, j := range t.J {

        if i != 0 {
            buff.WriteByte(',')
        }
        buff.Write([]byte(j))
    }
    buff.WriteByte(']')

    var buffer bytes.Buffer
    if err := json.Compact(&buffer, buff.Bytes()); err != nil {
        println(err.Error()) //error message: invalid character ',' looking for beginning of value
        return

    }

    w.Header().Set("Content-Type", contentTypeJSON)

    w.Write(buffer.Bytes())

}

func (myObj *myType) imports(pk string) error {

    cmd := exec.Command("go", "list", "-json", pk)
    stdout, _ := cmd.Output()

    pack[pk] = pk

    var deplist Import
    json.Unmarshal(stdout, &deplist)

    var newj json.RawMessage
    json.Unmarshal(stdout, &newj)
    myObj.J = append(myObj.J, newj)

    for _, imp := range deplist.Imports {

        if _, ok := pack[imp]; !ok {

            myObj.imports(imp) //recursive call to get the imports of the imports etc

        }
    }

    return nil

}

Answer 1

First, as has been commented, are you sure you can't use the go/build package directly rather than running go list?

I Wouldn't use println (or fmt.Println) inside HTTP handlers. It's much better to use log.Println and/or get the error into the ResponseWriter. Also, it's a good idea to wrap your ListenAndServe call with log.Fatal.

When printing/logging error values you can just use err, no need to have err.Error().

Further, when you actually want to do something more detailed than just reporting/logging the error message you can look at it's type and other info. For example, log.Printf("verbose error info: %#v", err) gives:

&json.SyntaxError{msg:"invalid character ',' looking for beginning of value", Offset:0}

I tried this because I know the json package returns various error types with additional info and I was hoping the offset value would be of help. If it had been then something like this might have been helpful:

if err := json.Compact(…) {
    if err != nil {
        log.Println("json.Compact:", err)
        if serr, ok := err.(*json.SyntaxError); ok {
            log.Println("Occurred at offset:", serr.Offset)
            // … something to show the data in buff around that offset …
        }
    }
}

But offset zero isn't helpful :(

So although this doesn't identify you problem hopefully it can be of some help to your further investigation.

Edit:

So after adding:

log.Println("Write file:", ioutil.WriteFile("data.json", buff.Bytes(), 0600))

to the above error handling block I then ran a JSON validator on the resultant file and it identified this piece:

        "XTestImports": [
                "io",
                "log",
                "net"
        ]
},,{
        "Dir": "/usr/local/go/src/mime",
        "ImportPath": "mime",
        "Name": "mime",

Note the double ,,.

That should tell you whete the error in your code is. But if not, you need to skip empty entries, either when processing t.J or when you build it. The later is better and just involves:

    if len(newj) > 0 {
        myObj.J = append(myObj.J, newj)
    }

(where btw you don't check for errors from json.Unmarshal so it's not clear if that is supposed to ever be empty or if it's empty due to a preceeding error. Never ignore error returns!)

Answer 2

I also encountered the same error message in a Go program, but the error message was within the HTTP response error, in HTML format when my HTTP response parser expected JSON.

For me, the solution was to change my request to include setting the Content-Type header to application/json. How you do this depends on which http client library you happen to be using; if you have access to the http.Header core type, you can set the header with .Set(...).

I realize the scope of this fix for me may not apply to the original question, but I came here first after googling and thought this would help others, since the message was not particularly obvious at first glance. The hint is that the invalid < character is the first HTML character in the error/response, which is likely the result of the request type not being set to application/json, thus the server responds with a non JSON response.

Answer 3

For me the issue was I was trying to parse the already parsed JSON.