ioutil.ReadAll(response.Body) blocks forever - Golang

Question

tr := &http.Transport{
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client := &http.Client{Transport: tr}
response, err := client.Get(link)
if err != nil {
    fmt.Println(err)
}
defer response.Body.Close()

//block forever at the next line
content, _ = ioutil.ReadAll(response.Body)

The above is my code to read content from a webpage which resides in a loop. I found sometimes the line ioutil.ReadAll(response.Body) will block forever. This happens randomly, however, it almost always happens on this webpage: http://xkcd.com/55 . It's very interesting that when I do curl http://xkcd.com/55, it returns nothing, however, wget http://xkcd.com/55 returns the whole webpage.

Answer 1

Additionally, avoid read response Body in ReadAll without memory/buffer limits control, example:

googleResponse := GoogleResponse{}
err = json.NewDecoder(io.LimitReader(resp.Body, MAX_MEMORY)).Decode(&googleResponse)
if err != nil {
    return nil, err
}

Read more about it in good blog posts:
Crossing Streams: a Love Letter to io.Reader by Jason Moiron
ioutil.ReadAll(httpResponse.Body) memory consumption
Golang Slices And The Case Of The Missing Memory