Lookup [HOST]: no such host error in Go

Question

I have this test program which will fetch url parallel, but when I increase the parallel number to about 1040, I start to get lookup www.httpbin.org: no such host error.

After some Google, I found others say that not close the response will cause the problem, but I do close that with res.Body.Close().

What's the problem here? thanks very much.

package main

import (
    "fmt"
    "net/http"
    "io/ioutil"
)

func get(url string) ([]byte, error) {

    client := &http.Client{}
    req, _ := http.NewRequest("GET", url, nil)

    res, err := client.Do(req)

    if err != nil {
        fmt.Println(err)
        return nil, err
    } 

    bytes, read_err := ioutil.ReadAll(res.Body)
    res.Body.Close()

    fmt.Println(bytes)

    return bytes, read_err
}

func main() {
    for i := 0; i < 1040; i++ {
        go get(fmt.Sprintf("http://www.httpbin.org/get?a=%d", i))
    }
}

Answer 1

well technically your process is limited (by the Kernel) to about 1000 open file descriptors. Depending on the context you might need to increase this number.

In your shell run (note the last line):

$ ulimit -a
-t: cpu time (seconds)         unlimited
-f: file size (blocks)         unlimited
-d: data seg size (kbytes)     unlimited
-s: stack size (kbytes)        8192
-c: core file size (blocks)    0
-v: address space (kb)         unlimited
-l: locked-in-memory size (kb) unlimited
-u: processes                  709
-n: file descriptors           2560

To increase (temporarly):

$ ulimit -n 5000
(no output)

Then verify the fd limit:

$ ulimit -n
5000