How to print the bytes while the file is being downloaded ? -golang

Question

I'm wondering if it's possible to count and print the number of bytes downloaded while the file is being downloaded.

out, err := os.Create("file.txt")
defer out.Close()
if err != nil {
    fmt.Println(fmt.Sprint(err))
    panic(err)
}
resp, err := http.Get("http://example.com/zip")
defer resp.Body.Close()
if err != nil {
    fmt.Println(fmt.Sprint(err))
    panic(err)
}

n, er := io.Copy(out, resp.Body)
if er != nil {
    fmt.Println(fmt.Sprint(err))
}
fmt.Println(n, "bytes ")

Answer 1

The stdlib now provides something like jimt's PassThru: io.TeeReader. It helps simplify things a bit:

// WriteCounter counts the number of bytes written to it.
type WriteCounter struct {
    Total int64 // Total # of bytes transferred
}

// Write implements the io.Writer interface.  
// 
// Always completes and never returns an error.
func (wc *WriteCounter) Write(p []byte) (int, error) {
    n := len(p)
    wc.Total += int64(n)
    fmt.Printf("Read %d bytes for a total of %d\n", n, wc.Total)
    return n, nil
}

func main() {

    // ...    

    // Wrap it with our custom io.Reader.
    src = io.TeeReader(src, &WriteCounter{})

    // ...
}

playground

Answer 2

If I understand you correctly, you wish to display the number of bytes read, while the data is transferring. Presumably to maintain some kind of a progress bar or something. In which case, you can use Go's compositional data structures to wrap the reader or writer in a custom io.Reader or io.Writer implementation.

It simply forwards the respective Read or Write call to the underlying stream, while doing some additional work with the (int, error) values returned by them. Here is an example you can run on the Go playground.

package main

import (
    "bytes"
    "fmt"
    "io"
    "os"
    "strings"
)

// PassThru wraps an existing io.Reader.
//
// It simply forwards the Read() call, while displaying
// the results from individual calls to it.
type PassThru struct {
    io.Reader
    total int64 // Total # of bytes transferred
}

// Read 'overrides' the underlying io.Reader's Read method.
// This is the one that will be called by io.Copy(). We simply
// use it to keep track of byte counts and then forward the call.
func (pt *PassThru) Read(p []byte) (int, error) {
    n, err := pt.Reader.Read(p)
    pt.total += int64(n)

    if err == nil {
        fmt.Println("Read", n, "bytes for a total of", pt.total)
    }

    return n, err
}

func main() {
    var src io.Reader    // Source file/url/etc
    var dst bytes.Buffer // Destination file/buffer/etc

    // Create some random input data.
    src = bytes.NewBufferString(strings.Repeat("Some random input data", 1000))

    // Wrap it with our custom io.Reader.
    src = &PassThru{Reader: src}

    count, err := io.Copy(&dst, src)
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

    fmt.Println("Transferred", count, "bytes")
}

The output it generates is this:

Read 512 bytes for a total of 512
Read 1024 bytes for a total of 1536
Read 2048 bytes for a total of 3584
Read 4096 bytes for a total of 7680
Read 8192 bytes for a total of 15872
Read 6128 bytes for a total of 22000
Transferred 22000 bytes