Detect gzip encoding to manually decompress response, but 'Content-Encoding' header missing

Question

I am using net/http library in 'Go' to make an HTTP GET request. In the response, i get 12 headers. But when i run the exact same query through postman, i get 16 headers. One of those missing is 'Content-Encoding'. I understand this must be a CORS issue.

But since i have not set the header Accept-Encoding: gzip in my request, and i am still getting the gzip encoding in response, the Go transport is not automatically decompressing the response for me. So, i need to be able to manually detect the encoding and then decompress it. But, i cannot detect if the 'Content-Encoding' header is missing in the response.

Here is my code where i try to do this:

func calcDistanceAndDurationWithUberApi(originLat float64, originLon float64, destinationLat float64, destinationLon float64) (float64, float64, error) {

    endpoint := "https://api.uber.com/v1.2/estimates/price"
    parameters := fmt.Sprintf("?start_latitude=%v&start_longitude=%v&end_latitude=%v&end_longitude=%v", originLat, originLon, destinationLat, destinationLon)

    req, err := http.NewRequest("GET", endpoint + parameters, nil)
    if err != nil {
        return 0, 0, err
    }

    req.Header.Add("Authorization", "Token " + getUberApiKey())
    req.Header.Add("Accept-Language", "en_US")
    req.Header.Add("Content-Type", "application/json")

    httpClient := &http.Client{}
    resp, err := httpClient.Do(req)
    if err != nil {
        return 0, 0, err
    }
    if resp.StatusCode != 200 {
        return 0, 0, errors.NotFound("Response: %v", resp.StatusCode)
    }
    defer resp.Body.Close()

    pretty.Println("- REQUEST: ")
    pretty.Println(req)

    // Check if server sent gzipped response. Decompress if yes.
    var respReader io.ReadCloser
    switch resp.Header.Get("Content-Encoding") {
    case "gzip":
        fmt.Println("Content-Encoding is gzip")
        respReader, err = gzip.NewReader(resp.Body)
        defer respReader.Close()
    default:
        fmt.Println("Content-Encoding is Not gzip")
        respReader = resp.Body
    }

    pretty.Println("- RESPONSE HEADER: ")
    pretty.Println(resp.Header)

    pretty.Println("- RESPONSE BODY: ")
    pretty.Println(respReader)

    return 0, 0, nil
}

The response status is '200 OK'. Here is the output (Response):

- RESPONSE HEADER: 
http.Header{
    "Content-Language":          {"en"},
    "Cache-Control":             {"max-age=0"},
    "X-Uber-App":                {"uberex-nonsandbox", "optimus"},
    "Strict-Transport-Security": {"max-age=604800", "max-age=2592000"},
    "X-Content-Type-Options":    {"nosniff"},
    "Date":                      {"Fri, 19 May 2017 07:52:17 GMT"},
    "Content-Geo-System":        {"wgs-84"},
    "Connection":                {"keep-alive"},
    "X-Frame-Options":           {"SAMEORIGIN"},
    "X-Xss-Protection":          {"1; mode=block"},
    "Server":                    {"nginx"},
    "Content-Type":              {"application/json"},
}
- RESPONSE BODY: 
&http.gzipReader{
body: &http.bodyEOFSignal{
    body: &http.body{
        src: &internal.chunkedReader{
            r:  &bufio.Reader{
                buf: {0x48, 0x54, .......... }

Answer 1

I gave in to the stubbornness of the uber api and added another request header, req.Header.Add("Accept-Encoding", "gzip").

Now i am getting the response header "Content-Encoding": "gzip", although i am still getting an undecipherable response body, but that's beyond the scope of this question.

Answer 2

If you don't disable compression [1], and you don't manually request compression with Accept-Encoding: gzip, then what I call "automatic mode" is used. With automatic mode, Go automatically adds Accept-Encoding: gzip, then if server responds Content-Encoding: gzip, Go wrap the response body in a Gzip reader, and removes the Content-Encoding and Content-Length response headers [2]. I disagree with this practice, as the end user is essentially being lied to about what the true response was. Contrast this with cURL, which gives you the pure response, regardless of what you do:

PS C:\> curl -v --compressed https://github.com/manifest.json
< content-encoding: gzip
< content-length: 345

To deal with this, I wrote a wrapper for http.Transport:

package mech

import (
   "compress/gzip"
   "io"
   "net/http"
   "strings"
)

type Transport struct { http.Transport }

func (t Transport) RoundTrip(req *http.Request) (*http.Response, error) {
   if !t.DisableCompression {
      req.Header.Set("Accept-Encoding", "gzip")
   }
   res, err := t.Transport.RoundTrip(req)
   if err != nil {
      return nil, err
   }
   if strings.EqualFold(res.Header.Get("Content-Encoding"), "gzip") {
      gz, err := gzip.NewReader(res.Body)
      if err != nil {
         return nil, err
      }
      res.Body = readCloser{gz, res.Body}
   }
   return res, nil
}

type readCloser struct {
   io.Reader
   io.Closer
}

1. https://golang.org/pkg/net/http#Transport.DisableCompression
2. https://github.com/golang/go/blob/go1.16.5/src/net/http/transport.go#L2186-L2192