I am building a tool in Go that needs to make a very large number of simultaneous HTTP requests to many different servers. My initial prototype in Python had no problem doing a few hundred simultaneous requests.
However, I have found that in Go this almost always results in a Get http://www.google.com: dial tcp 216.58.205.228:80: i/o timeout
for some if the number of simultaneous requests exceeds ~30-40.
I've tested on macOS, openSUSE, different hardware, in different networks and with different domain lists, and changing the DNS server as described in other Stackoverflow answers does not work either.
The interesting thing is that the failed requests do not even produce a packet, as can be seen when checking with Wireshark.
Is there anything that I am doing wrong or is that a bug in Go?
Minimum reproducible program below:
package main
import (
"fmt"
"net/http"
"sync"
)
func main() {
domains := []string{/* large domain list here, eg from https://moz.com/top500 */}
limiter := make(chan string, 50) // Limits simultaneous requests
wg := sync.WaitGroup{} // Needed to not prematurely exit before all requests have been finished
for i, domain := range domains {
wg.Add(1)
limiter <- domain
go func(i int, domain string) {
defer func() { <-limiter }()
defer wg.Done()
resp, err := http.Get("http://"+domain)
if err != nil {
fmt.Printf("%d %s failed: %s\n", i, domain, err)
return
}
fmt.Printf("%d %s: %s\n", i, domain, resp.Status)
}(i, domain)
}
wg.Wait()
}
Two particular error messages are happening, a net.DNSError
that does not make any sense and a non-descript poll.TimeoutError
:
&url.Error{Op:"Get", URL:"http://harvard.edu", Err:(*net.OpError)(0xc00022a460)}
&net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), Addr:net.Addr(nil), Err:(*net.DNSError)(0xc000aca200)}
&net.DNSError{Err:"no such host", Name:"harvard.edu", Server:"", IsTimeout:false, IsTemporary:false}
&url.Error{Op:"Get", URL:"http://latimes.com", Err:(*net.OpError)(0xc000d92730)}
&net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), Addr:net.Addr(nil), Err:(*poll.TimeoutError)(0x14779a0)}
&poll.TimeoutError{}
Update:
Running the requests with a seperate http.Client
as well as http.Transport
and net.Dialer
does not make any difference as can be seen when running code from this playground.
I think many of your net.DNSError
s are actually too many open files
errors in disguise. You can see this by running your sample code with the netgo
tag (recommendation from here) (go run -tags netgo main.go
) which will emit errors like:
…dial tcp: lookup buzzfeed.com on 192.168.1.1:53: dial udp 192.168.1.1:53: socket: too many open files
instead of
…dial tcp: lookup buzzfeed.com: no such host
Make sure you're closing the request's response body (resp.Body.Close()
). You can find more about this specific problem at What's the best way to handle "too many open files"? and How to set ulimit -n from a golang program?. (On my machine (macOS), increasing file limits manually seemed to help, but I don't think it's a good solution since it doesn't really scale, and I'm not sure how many open files you'd need overall.)
As suggested by @liam-kelly, I think the i/o timeout
error is coming from a DNS server or some other security mechanism. Setting a custom (bad) DNS server IP gives me the same error.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With