How to receive HTTP Response for streaming

Question

When throwing an HTTP Request with Go and receiving a Response, I want to receive a response while streaming, considering the case where the ResponseBody is huge (1 GB or more).

resp, err: = http.Client.Do(req)

In this case, if the body is huge, I can not read the Header and I do not know the state of Response. Is there any solution?

Answer 1

(Edit: If you're unable to get the "Content-length" header from the response, it is possible that the web service you're hitting doesn't return that header. In such a case, there's no way to know the length of the response body without reading it completely. You can simulate that in the following example by removing the line that sets the Content-length header in the response.)

The standard Go net/http package handles large responses very well. Here's a self contained example to demonstrate:

// Start a mock HTTP server that returns 2GB of data in the response. Make a
// HTTP request to this server and print the amount of data read from the
// response.
package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
    "strings"
    "time"
)

const oneMB = 1024 * 1024
const oneGB = 1024 * oneMB
const responseSize = 2 * oneGB

const serverAddr = "localhost:9999"

func startServer() {
    // Mock HTTP server that always returns 2GB of data
    go http.ListenAndServe(serverAddr, http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
        w.Header().Set("Content-length", fmt.Sprintf("%d", responseSize))

        // 1MB buffer that'll be copied multiple times to the response
        buf := []byte(strings.Repeat("x", oneMB))

        for i := 0; i < responseSize/len(buf); i++ {
            if _, err := w.Write(buf); err != nil {
                log.Fatal("Failed to write to response. Error: ", err.Error())
            }
        }
    }))

    // Some grace period for the server to start
    time.Sleep(100 * time.Millisecond)
}

func main() {
    startServer()

    // HTTP client
    req, err := http.NewRequest("GET", "http://"+serverAddr, nil)
    if err != nil {
        log.Fatal("Error creating HTTP request: ", err.Error())
    }

    client := http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        log.Fatal("Error making HTTP request: ", err.Error())
    }

    // Read the response header
    fmt.Println("Response: Content-length:", resp.Header.Get("Content-length"))

    bytesRead := 0
    buf := make([]byte, oneMB)

    // Read the response body
    for {
        n, err := resp.Body.Read(buf)
        bytesRead += n

        if err == io.EOF {
            break
        }

        if err != nil {
            log.Fatal("Error reading HTTP response: ", err.Error())
        }
    }

    fmt.Println("Response: Read", bytesRead, "bytes")
}

You wouldn't want to read the entire response in memory if it's too large. Write it to a temporary file instead and then process that.

If instead you're looking for options to do this reliably when the network isn't very reliable, look for "HTTP range requests" using which you can resume partially downloaded data.