How to unzip io.ReadCloser?

Question

I fetch a data from http service and I want to unzip it on the fly. Here is my current approach:

resp, err := http.Get(url)
if err != nil {
    logger.Fatalf("Can't fatch %s data. %v", url, err)
}
logger.Info("Fetched data from %s", url)
content_zipped, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
    logger.Fatal(err)
}
zip_reader, err := zip.NewReader(bytes.NewReader(content_zipped), int64(len(content_zipped)))

Is there any way to unzip the resp.Body without reading all the content at once (6. line)? I mean to stream the bytes.

Answer 1

Are you looking for something like below:

var ioReader io.Reader
buff := bytes.NewBuffer([]byte{})
size, err := io.Copy(buff, ioReader)
if err != nil {
    return err
}

reader := bytes.NewReader(buff.Bytes())

// Open a zip archive for reading.
zipReader, err := zip.NewReader(reader, size)

Answer 2

Zip archives require random access for reading, so it's hard to stream bytes. In particular, see the source for zip.Reader.init here: http://golang.org/src/pkg/archive/zip/reader.go?s=1265:1323#L59 . The first thing it does is call readDirectoryEnd which reads from near the end of the file.

Can you use a different compression method (for example, gzip)? Then you can use gzip.NewReader(resp.Body) to stream the data.