Go illegal base64 data at input byte 0 when unmarshalling a string json

Question

I'm storing my data in a redis database and when I ask for it, I get a valid json back. The json looks like this:

{"Data":"hi","Hash":"000f7dcfca98450a0f405384db3878c1956cb98309e63cf2d0a963cff9f17260","PrevHash":"000daf177434acd55a3284787b793a3453c3d70eacdb9a84f5faed43adb2ff58","Nonce":8504,"TimeStamp":1611498968}

It's a valid json string (Go omits leading and traling quotes) but I get this error when unmarshalling. illegal base64 data at input byte 0

err = json.Unmarshal([]byte(item), &block)

type Block struct {
    Data      []byte
    Hash      []byte
    PrevHash  []byte
    Nonce     int
    TimeStamp int64
}

func (chain *BlockChain) AddBlock(data string) {
    var lastHash []byte

    item, err := chain.Database.Get(ctx, "lastHash").Result()
    Handle(err)
    lastHash = []byte(item)
    newBlock := createBlock(data, lastHash)

    _, err = chain.Database.Set(ctx, StrHash(newBlock.Hash), newBlock, 0).Result()
    Handle(err)
    _, err = chain.Database.Set(ctx, "lastHash", newBlock.Hash, 0).Result()
}

func (chain *BlockChain) Iterator() *BlockChainIterator {
    return &BlockChainIterator{
        CurrentHash: chain.LastHash,
        Database:    chain.Database,
    }
}

func (iterator *BlockChainIterator) Next() *Block {
    var block *Block

    item, err := iterator.Database.Get(ctx, StrHash(iterator.CurrentHash)).Result()
    
    Handle(err)
    err = json.Unmarshal([]byte(item), &block)

    Handle(err)
    iterator.CurrentHash = block.PrevHash

    return block

}

// -----------------
// Utility Functions
// -----------------

// TODO: Think about this. Wouldn't it be better to store everything as strings rather than converting it?
type jsonBlock struct {
    Data      string
    Hash      string
    PrevHash  string
    Nonce     int
    TimeStamp int64
}

// This function gets called automatically by go-redis
func (b *Block) MarshalBinary() ([]byte, error) {
    jb := jsonBlock{
        Data:      string(b.Data),
        Hash:      StrHash(b.Hash),
        PrevHash:  StrHash(b.PrevHash),
        Nonce:     b.Nonce,
        TimeStamp: b.TimeStamp,
    }
    return json.Marshal(jb)
}

Answer 1

The Hash value (and also PrevHash) in JSON is not the Base64 encoded form of the data, but rather a hexadecimal representation. If you try to unmarshal it into a Go value of type []byte, the encoding/json package assumes and try to interpret it as Base64 encoded data.

Same goes for the Data field: it should be of Go type string since it is not the Base64 encoded form, so change it to string.

Try to unmarshal Hash and PrevHash into fields of type string, and do hexa decoding using hex.DecodeString().

You may implement a custom unmarshaler to do this automatically.

A convenient way would be to create a HexData type that is a []byte but decodes from hex string:

type HexData []byte

func (h *HexData) UnmarshalJSON(data []byte) error {
    var s string
    if err := json.Unmarshal(data, &s); err != nil {
        return err
    }
    decoded, err := hex.DecodeString(s)
    if err != nil {
        return err
    }
    *h = HexData(decoded)
    return nil
}

Using it:

type Block struct {
    Data      string
    Hash      HexData
    PrevHash  HexData
    Nonce     int
    TimeStamp int64
}

func main() {
    var b Block
    if err := json.Unmarshal([]byte(src), &b); err != nil {
        panic(err)
    }
    fmt.Printf("%+v\n", b)
}

Output (try it on the Go Playground):

{Data:hi Hash:[0 15 125 207 202 152 69 10 15 64 83 132 219 56 120 193 149 108 185 131 9 230 60 242 208 169 99 207 249 241 114 96] PrevHash:[0 13 175 23 116 52 172 213 90 50 132 120 123 121 58 52 83 195 215 14 172 219 154 132 245 250 237 67 173 178 255 88] Nonce:8504 TimeStamp:1611498968}