net/http Transport Exceeding MaxConnsPerHost

Question

I have a simple program below which is a simple HTTP client and server. I'm testing if MaxConnsPerHost in http.Transport introduced in Go 1.11 is working as advertised. However, when I run the code more than 10-30 minutes, the ESTABLISHED connections slowly exceeds the set MaxConnsPerHost. Am I doing something wrong?

package main

import (
    "io/ioutil"
    "log"
    "net"
    "net/http"
    "time"
)

func main() {

    // Server
    //
    go func() {

        if err := http.ListenAndServe(":8081", http.HandlerFunc(
            func(w http.ResponseWriter, r *http.Request) {
                w.Write([]byte("hi"))
            },
        )); err != nil {
            log.Fatal(err)
        }

    }()

    // Client settings
    //
    c := &http.Client{
        Timeout: 30 * time.Second,
        Transport: &http.Transport{
            Proxy: http.ProxyFromEnvironment,
            DialContext: (&net.Dialer{
                Timeout:   30 * time.Second,
                KeepAlive: 30 * time.Second,
                DualStack: true,
            }).DialContext,
            MaxIdleConns:          100,
            IdleConnTimeout:       90 * time.Second,
            TLSHandshakeTimeout:   10 * time.Second,
            ExpectContinueTimeout: 1 * time.Second,
            MaxConnsPerHost:       50,
        },
    }

    // Worker loop
    //
    threads := 500
    for i := 0; i < threads; i++ {

        go func() {

            for {

                req, err := http.NewRequest("GET", "http://localhost:8081", nil)
                if err != nil {
                    log.Fatal(err)
                }

                res, err := c.Do(req)
                if err != nil {
                    log.Fatal(err)
                }

                if _, err := ioutil.ReadAll(res.Body); err != nil {
                    log.Fatal(err)
                }

                res.Body.Close()
            }

        }()
    }

    var done chan bool
    <-done
    log.Println("Done")

}

After running this for a long time, the ESTABLISHED connections reported by netstat is already exceeding 50.

P.S. We have an in issue in one of our services which horribly leaks ESTABLISHED connections even though we are properly closing the response's Body. It is currently built using Go 1.10 and I was hoping Go 1.11 MaxConnsPerHost would be a solution but it seems to also crack under heavy load.

Answer 1

Am I doing something wrong?

It appears this bug has been resolved in more recent go versions, I can not reproduce this with:

$ go version 
go version go1.17.1 linux/amd64

monitoring:

watch -n 3 'ss  -tn | grep :8081 | sort -k 3,3n | tee >(wc -l) | tail ;
            ps -eo pid,etime,cmd | grep "leaky[-]server"'

running the server:

go run leaky-server.go

sample output after 2.5 hours:

Every 3.0s: ss  -tn | grep :8081 | sor...  balmora: Sun Oct 31 12:38:52 2021

ESTAB 0      118                 [::1]:8081               [::1]:43814
ESTAB 0      118                 [::1]:8081               [::1]:43818
ESTAB 0      118                 [::1]:8081               [::1]:43822
ESTAB 0      118                 [::1]:8081               [::1]:43824
ESTAB 0      118                 [::1]:8081               [::1]:43830
ESTAB 0      118                 [::1]:8081               [::1]:43834
ESTAB 0      118                 [::1]:8081               [::1]:43836
ESTAB 0      118                 [::1]:8081               [::1]:43838
ESTAB 0      118                 [::1]:8081               [::1]:43844
100
 536182    02:30:03 go run leaky-server.go
 536269    02:30:03 /tmp/go-build2453595122/b001/exe/leaky-server