Limit bytes to read from HTTP response

Question

I need to read responses from user provided URLs

I don't want them to overload my server with links to huge files.

I want to read N bytes max and return an error if there are more bytes to read.

I can read N bytes, but how I detect, that file is incomplete (assuming corner cases when remote file is exactly N bytes long)?

Answer 1

Additionally to Peter's answer, there is a ready solution in the net/http package: http.MaxBytesReader():

func MaxBytesReader(w ResponseWriter, r io.ReadCloser, n int64) io.ReadCloser

MaxBytesReader is similar to io.LimitReader but is intended for limiting the size of incoming request bodies. In contrast to io.LimitReader, MaxBytesReader's result is a ReadCloser, returns a non-EOF error for a Read beyond the limit, and closes the underlying reader when its Close method is called.

Originally it was "designed" for limiting the size of incoming request bodies, but it can be used to limit incoming response bodies as well. For that, simply pass nil for the ResponseWriter parameter.

Example using it:

{
    body := ioutil.NopCloser(bytes.NewBuffer([]byte{0, 1, 2, 3, 4}))
    r := http.MaxBytesReader(nil, body, 4)
    buf, err := ioutil.ReadAll(r)
    fmt.Println("When body is large:", buf, err)
}

{
    body := ioutil.NopCloser(bytes.NewBuffer([]byte{0, 1, 2, 3, 4}))
    r := http.MaxBytesReader(nil, body, 5)
    buf, err := ioutil.ReadAll(r)
    fmt.Println("When body is exact (OK):", buf, err)
}

{
    body := ioutil.NopCloser(bytes.NewBuffer([]byte{0, 1, 2, 3, 4}))
    r := http.MaxBytesReader(nil, body, 6)
    buf, err := ioutil.ReadAll(r)
    fmt.Println("When body is small (OK):", buf, err)
}

Output (try it on the Go Playground):

When body is large: [0 1 2 3] http: request body too large
When body is exact (OK): [0 1 2 3 4] <nil>
When body is small (OK): [0 1 2 3 4] <nil>