How to handle HTTP timeout errors and accessing status codes in golang

Question

I have some code (see below) written in Go which is supposed to "fan-out" HTTP requests, and collate/aggregate the details back.

I'm new to golang and so expect me to be a nOOb and my knowledge to be limited

The output of the program is currently something like:

{
    "Status":"success",
    "Components":[
        {"Id":"foo","Status":200,"Body":"..."},
        {"Id":"bar","Status":200,"Body":"..."}, 
        {"Id":"baz","Status":404,"Body":"..."}, 
        ...
    ]
}

There is a local server running that is purposely slow (sleeps for 5 seconds and then returns a response). But I have other sites listed (see code below) that sometime trigger an error as well (if they error, then that's fine).

The problem I have at the moment is how best to handle these errors, and specifically the "timeout" related errors; in that I'm not sure how to recognise if a failure is a timeout or some other error?

At the moment I get a blanket error back all the time:

Get http://localhost:8080/pugs: read tcp 127.0.0.1:8080: use of closed network connection

Where http://localhost:8080/pugs will generally be the url that failed (hopefully by timeout!). But as you can see from the code (below), I'm not sure how to determine the error code is related to a timeout nor how to access the status code of the response (I'm currently just blanket setting it to 404 but obviously that's not right - if the server was to error I'd expect something like a 500 status code and obviously I'd like to reflect that in the aggregated response I send back).

The full code can be seen below. Any help appreciated.

    package main

    import (
            "encoding/json"
            "fmt"
            "io/ioutil"
            "net/http"
            "sync"
            "time"
    )

    type Component struct {
            Id  string `json:"id"`
            Url string `json:"url"`
    }

    type ComponentsList struct {
            Components []Component `json:"components"`
    }

    type ComponentResponse struct {
            Id     string
            Status int
            Body   string
    }

    type Result struct {
            Status     string
            Components []ComponentResponse
    }

    var overallStatus string = "success"

    func main() {
            var cr []ComponentResponse
            var c ComponentsList

            b := []byte(`{"components":[{"id":"local","url":"http://localhost:8080/pugs"},{"id":"google","url":"http://google.com/"},{"id":"integralist","url":"http://integralist.co.uk/"},{"id":"sloooow","url":"http://stevesouders.com/cuzillion/?c0=hj1hfff30_5_f&t=1439194716962"}]}`)

            json.Unmarshal(b, &c)

            var wg sync.WaitGroup

            timeout := time.Duration(1 * time.Second)
            client := http.Client{
                    Timeout: timeout,
            }

            for i, v := range c.Components {
                    wg.Add(1)

                    go func(i int, v Component) {
                            defer wg.Done()

                            resp, err := client.Get(v.Url)

                            if err != nil {
                                fmt.Printf("Problem getting the response: %s\n", err)

                                cr = append(cr, ComponentResponse{
                                    v.Id,
                                    404,
                                    err.Error(),
                                })
                            } else {
                                    defer resp.Body.Close()
                                    contents, err := ioutil.ReadAll(resp.Body)
                                    if err != nil {
                                            fmt.Printf("Problem reading the body: %s\n", err)
                                    }

                                    cr = append(cr, ComponentResponse{
                                            v.Id,
                                            resp.StatusCode,
                                            string(contents),
                                    })
                            }
                    }(i, v)
            }
            wg.Wait()

            j, err := json.Marshal(Result{overallStatus, cr})
            if err != nil {
                    fmt.Printf("Problem converting to JSON: %s\n", err)
                    return
            }

            fmt.Println(string(j))
    }

Answer 1

If you want to fan out then aggregate results and you want specific timeout behavior the net/http package isn't giving you, then you may want to use goroutines and channels.

I just watched this video today and it will walk you through exactly those scenarios using the concurrency features of Go. Plus, the speaker Rob Pike is quite the authority -- he explains it much better than I could.

https://www.youtube.com/watch?v=f6kdp27TYZs

Answer 2

I am adding this for completes, as the correct answer was provided by Dave C in the comments of the accepted answer.

We can try to cast the error to a net.Error and check if it is a timeout.

resp, err := client.Get(url)
if err != nil {
    // if there is an error check if its a timeout error
    if e, ok := err.(net.Error); ok && e.Timeout() {
        // handle timeout
           return
    } 
    // otherwise handle other types of error
}