In Go, how do I close a long running read?

Question

It doesn't seem possible to have two way communication via channels with a goroutine which is performing file operations, unless you block the channel communication on the file operations. How can I work around the limits this imposes?

Another way to phrase this question...

If I have a loop similar to the following running in a goroutine, how can I tell it to close the connection and exit without blocking on the next Read?

func readLines(response *http.Response, outgoing chan string) error {
    defer response.Body.Close()
    reader := bufio.NewReader(response.Body)

    for {
        line, err := reader.ReadString('\n')
        if err != nil {
            return err
        }
        outgoing <- line
    }
}

It's not possible for it to read from a channel that tells it when to close down because it's blocking on the network reads (in my case, that can take hours).

It doesn't appear to be safe to simply call Close() from outside the goroutine, since the Read/Close methods don't appear to be fully thread safe.

I could simply put a lock around references to response.Body that used inside/outside the routine, but would cause the external code to block until a pending read completes, and I specifically want to be able to interrupt an in-progress read.

Answer 1

To address this scenario, several io.ReadCloser implementations in the standard library support concurrent calls to Read and Close where Close interrupts an active Read.

The response body reader created by net/http Transport is one of those implementations. It is safe to concurrently call Read and Close on the response body.

You can also interrupt an active Read on the response body by calling the Transport CancelRequest method.

Here's how implement cancel using close on the body:

func readLines(response *http.Response, outgoing chan string, done chan struct{}) error {
    cancel := make(chan struct{})
    go func() {
       select {
       case <-done:
          response.Body.Close()
       case <-cancel:
          return
    }()

    defer response.Body.Close()
    defer close(cancel) // ensure that goroutine exits

    reader := bufio.NewReader(response.Body)
    for {
        line, err := reader.ReadString('\n')
        if err != nil {
            return err
        }
        outgoing <- line
    }
}

Calling close(done) from another goroutine will cancel reads on the body.