Garbled string while do http request in Go

Question

I met a Garbled string question when do Get request in Go, the code is:

req , err:= http.NewRequest(httpMethod, url,strings.NewReader(""))
req.Header.Add("Accept","application/json")
resp, err := http.DefaultClient.Do(req)
body,err := ioutil.ReadAll(resp.Body)
ret := string(body)
log.Warningf("ret: %+v", ret)

if the ret contains only english, it's correct, if contains Chinese, it has garbled string, how to solve this problem, thanks all!

Answer 1

In my case website did not respond with the charset within a Content-Type and did not replied with requested: req.Header.Add("Accept-Charset", "utf-8")

I opended up file in Visual Studio Code and have been switching encoding to find out which one works best "Reopen With Encoding".

Once i figured out which encoding it was i simply used function:

dec := charmap.Windows1250.NewDecoder()
output, _ := dec.Bytes(body)

from: "golang.org/x/text/encoding/charmap"

Full code example:

package main

import (
    "fmt"
    "net/http"
    "io/ioutil"
    "golang.org/x/text/encoding/charmap"
)

func main() {

    client := &http.Client{}

    req, err := http.NewRequest("GET", "example.com", nil)

    if err != nil {
        fmt.Println(err)
        return
    }

    resp, err := client.Do(req)

    if err != nil {
        fmt.Println(err)
        return
    }

    defer resp.Body.Close()

    body, err := ioutil.ReadAll(resp.Body)

    if err != nil {
        fmt.Println(err)
        return
    }

    dec := charmap.Windows1250.NewDecoder()

    output, err:= dec.Bytes(body)

    if err != nil {
        fmt.Println(err)
        return
    }

    // do something with output

}

Answer 2

Go strings can hold any type of characters, but when printing them the chars are interpreted as utf-8.

You can try adding:

req.Header.Add("Accept-Charset","utf-8")

If that does not work, you can try using this package to convert from whatever charset it is to utf-8:

https://godoc.org/golang.org/x/text/encoding

The charset depends on the page you are requesting. If it is html, the charset is sometimes specified like this in the response headers:

Content-Type: text/html; charset=utf-8

So you need to figure out what the charset is.