Reading http.Response Body stream

Question

I'm sending a http request and getting a response in text/xml format. I don't need to parse the XML, just trying to return it as a string for testing. But it seems pretty slow.

The XML response is 278kb, but it can take about 1.7 but sometimes spike to 4.5seconds just to read the response body. I can rerun this over and over and it can vary wildly. I'm not sure what's causing it as i'm new to Go.

I feel like the ioutil.ReadAll() function is letting me down. But i've also tried bufio.NewReader and reader.ReadBytes. All seem to be just as slow.

this is what my function looks like:

client := &http.Client{}
req, err := http.NewRequest("POST", url, strings.NewReader(requestBody))
if err != nil {
    fmt.Println("New Request Error", err)
}

// Set Headers
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("Content-Encoding", "gzip")
req.Header.Set("Accept-Encoding", "gzip, deflate")

defer req.Body.Close()

// Get Response
startRes := time.Now()
response, err := client.Do(req)
if response != nil {
    defer response.Body.Close()
}
if err != nil {
    fmt.Println("POST Error", err)
}
endRes := time.Now()
fmt.Println("Response took", endRes.Sub(startRes))

// Read Response Body
startRead := time.Now()
body, readErr := ioutil.ReadAll(response.Body)
if readErr != nil {
    fmt.Println("body error:", readErr)
    return nil, readErr
}

endRead := time.Now()
fmt.Println("Read response took:", endRead.Sub(startRead))

The output of these println's look like this:

> go run main.go
 Response took 5.230523s
 Read response took: 1.7551755s

Is there a way to make this code more efficient? I'm assuming the 1.7s read time for a 278kb file is due to network lag, right?

Answer 1

I recommend allocating a byte.Buffer and reading the body yourself. The code at ioutil.ReadAll creates a byte.Buffer at an initial capacity of Byte.MinRead (512 bytes).

At 278KB, Buffer needs to expand, reallocate and recopy the returned byte array 10 times which is most processing time is being spent (for things you can control).

Untested code to try below:

buf := bytes.NewBuffer(make([]byte, 0, response.ContentLength))
_, readErr = buf.ReadFrom(response.Body)
body := buf.Bytes()