Reading gzipped HTTP response in Go

Question

I am trying to read a gzipped HTTP response with Go! but I always get the following error message :

panic: gzip: invalid header [...] stack trace [...] 

If I run "curl -H "Accept-Encoding: gzip" http://foo.com/ | gunzip -" I get the response correctly gunzipped. I also double checked with ngrep and the pair Accept-Encoding/Content-Encoding is correctly sent/returned.

If I create a file with some dummy content and gzip it, I can read it from my Go! program.

The program I used for testing:

package main  import (     "io"     //"os"     "fmt"     "compress/gzip"     "net/http" )  func main() {     /* This works fine     f, _ := os.Open("/tmp/test.gz")     defer f.Close()     reader, err := gzip.NewReader(f)     */      // This does not :/     resp, _ := http.Get("http://foo.com/")     defer resp.Body.Close()     reader, err := gzip.NewReader(resp.Body)      if err != nil { panic(err) }      buff := make([]byte, 1024)     for {         n, err := reader.Read(buff)          if err != nil && err != io.EOF {             panic(err)         }          if n == 0 {             break         }     }      s := fmt.Sprintf("%s", buff)     fmt.Println(s) } 

Have I overlooked something ?

Answer 1

EDIT: The following is an example of manually handling compression. If you don't set the header, the default Transport will do it for you and then decompress while you read the response.Body.

client := new(http.Client)  request, err := http.NewRequest("GET", "http://stackoverflow.com", nil) request.Header.Add("Accept-Encoding", "gzip")  response, err := client.Do(request) defer response.Body.Close()  // Check that the server actually sent compressed data var reader io.ReadCloser switch response.Header.Get("Content-Encoding") { case "gzip":     reader, err = gzip.NewReader(response.Body)     defer reader.Close() default:     reader = response.Body }  io.Copy(os.Stdout, reader) // print html to standard out 

Error handling removed for brevity. I kept the defers.

Answer 2

net/http#Transport handles gzip compressed responses. You don't have to do anything special.

Take a look at the DisableCompression option, here.