Process Management for the Go Webserver

Question

I'm a new Go programmer, coming from the world of web application and service development. Apologies is this is a herp de-derp question, but my googling around for an answer hasn't found anything. Also, this is borderline Server Fault territory, but since I'm more interested in the APIs/programmatic interfaces I'm asking here.

I've written a small go program using the net/http package's built-in web server. I'm getting ready to deploy to production, but I'm a little unclear on the process of model go Go's web-server and how I should deploy.

Specifically -- in the environments I'm used to (PHP, Ruby, Python) we have a web server (Apache, Nginx, etc.) sitting in front of our application, and we configure these web servers to use a certain number of worker processes/threads, and configure how many individual HTTP(S) connections each thread should process.

I haven't been able to find information on how Go's web-server handles this, or practical information on how to scale/plan-for-scale for a Go web server.

i.e. -- if I have a simple program running, ready to handle an HTTP request

func main() {
   http.HandleFunc("/", processRequest)
   http.ListenAndServe(":8000", nil)    
}

how many connections will HandleFunc try to handle at once? Or will it start blocking when a connection opens, and only serve the next connection once the connection closes?

Or should I just not worry about this and jam everything into a go routine? But if I do that how do I prevent the system from becoming bogged down by too many threads of execution?

I'm basically trying to

1. Understand the process mode of the go web server
2. Find the built-in features of go for tweaking this, and/or whatever standard package folks are using

Like I said, I'm very new to go, so if I'm completely missing the plot on this please let me know!

Answer 1

Tweaking / configuring the HTTP server

The type that implements the HTTP server is http.Server. If you don't create an http.Server yourself e.g. because you call the http.ListenAndServe() function, that creates an http.Server under the hood for you:

func ListenAndServe(addr string, handler Handler) error {
    server := &Server{Addr: addr, Handler: handler}
    return server.ListenAndServe()
}

So if you want to tweek / customize the HTTP server, then create one yourself and call its Server.ListenAndServe() method yourself. http.Server is a struct, its zero value is a valid configuration. See its doc what fields it has and so what you can tweak / configure.

The "Process Management" of the HTTP server is documented at Server.Serve():

Serve accepts incoming connections on the Listener l, creating a new service goroutine for each. The service goroutines read requests and then call srv.Handler to reply to them. Serve always returns a non-nil error.

So each incoming HTTP request is handled in its new goroutine, meaning they are served concurrently. Unfortunately the API does not document any way to jump in and change how this works.

And looking at the current implementation (Go 1.6.2), there is also no undocumented way to do that. server.go, currently line #2107-2139:

2107    func (srv *Server) Serve(l net.Listener) error {
2108        defer l.Close()
2109        if fn := testHookServerServe; fn != nil {
2110            fn(srv, l)
2111        }
2112        var tempDelay time.Duration // how long to sleep on accept failure
2113        if err := srv.setupHTTP2(); err != nil {
2114            return err
2115        }
2116        for {
2117            rw, e := l.Accept()
2118            if e != nil {
2119                if ne, ok := e.(net.Error); ok && ne.Temporary() {
2120                    if tempDelay == 0 {
2121                        tempDelay = 5 * time.Millisecond
2122                    } else {
2123                        tempDelay *= 2
2124                    }
2125                    if max := 1 * time.Second; tempDelay > max {
2126                        tempDelay = max
2127                    }
2128                    srv.logf("http: Accept error: %v; retrying in %v", e, tempDelay)
2129                    time.Sleep(tempDelay)
2130                    continue
2131                }
2132                return e
2133            }
2134            tempDelay = 0
2135            c := srv.newConn(rw)
2136            c.setState(c.rwc, StateNew) // before Serve can return
2137            go c.serve()
2138        }
2139    }

As you can see in line #2137, the connection is served unconditionally on a new goroutine, so there's nothing you can do about that.

Limiting the "worker" goroutines

If you want to limit the number of goroutines serving requests, you can still do it.

You may limit them on multiple levels. For limiting on the listener level, see Darigaaz's answer. To limit on the Handler level, read on.

For example you could insert a code to each of your http.Handler or handler functions (http.HandlerFunc) which only proceeds if the number of concurrent request serving goroutines are less than a specified limit.

There are numerous constructs to such limiting-synchronization code. One example could be: create a buffered channel with capacity being your desired limit. Each handler should first send a value on this channel, and then do the work. When the handler returns, it must receive a value from the channel: so it's best done in a deferred function (not to forget to "clean" itself).

If the buffer is full, a new request attempting to send on the channel will block: wait until a request finishes its work.

Note that you don't have to inject this limiting code to all your handlers, you may use a "middleware" pattern, a new handler type which wraps your handlers, does this limiting-synchronization job, and calls the wrapped handler in the middle of it.

The advantage of limiting in the handler (as opposed to limiting in Listeners) is that in the handler we know what the handler does, so we can do selective limiting (e.g. we may choose to limit certain requests such as database operations, and not to limit others like serving static resources) or we can create multiple, distinct limit groups arbitrarily to our needs (e.g. limit concurrent db requests to 10 max, limit static requests to 100 max, limit heavy computational requests to 3 max) etc. We can also easily realize limitations like unlimited (or high limit) for logged-in / paying users, and low limit for anonymous / non-paying users.

Also note that you can even do the rate-limiting in a single place, without using middlewares. Create a "main handler", and pass that to http.ListenAndServe() (or Server.ListenAndServe()). In this main handler do the rate limiting (e.g. using a buffered channel as mentioned above), and simply forward the call to the http.ServeMux you're using.

Here's a simple example which uses http.ListenAndServe() and the default multiplexer of the http package (http.DefaultServeMux) for demonstration. It limits concurrent requests to 2:

func fooHandler(w http.ResponseWriter, r *http.Request) {
    log.Println("Foo called...")
    time.Sleep(3 * time.Second)
    w.Write([]byte("I'm Foo"))
    log.Println("Foo ended.")
}

func barHandler(w http.ResponseWriter, r *http.Request) {
    log.Println("Bar called...")
    time.Sleep(3 * time.Second)
    w.Write([]byte("I'm Bar"))
    log.Println("Bar ended.")
}

var ch = make(chan struct{}, 2) // 2 concurrent requests

func mainHandler(w http.ResponseWriter, r *http.Request) {
    ch <- struct{}{}
    defer func() {
        <-ch
    }()

    http.DefaultServeMux.ServeHTTP(w, r)
}

func main() {
    http.HandleFunc("/foo", fooHandler)
    http.HandleFunc("/bar", barHandler)

    panic(http.ListenAndServe(":8080", http.HandlerFunc(mainHandler)))
}

Deployment

Web applications written in Go do not require external servers to control processes, as the Go webserver itself handles requests concurrently.

So you may start your webserver written in Go as-is: the Go webserver is production ready.

You may of course use other servers for additional tasks if you wish so (e.g. HTTPS handling, authentication / authorization, routing, load balancing between multiple servers).

Answer 2

ListenAndServe starts an HTTP server with a given address and handler. The handler is usually nil, which means to use DefaultServeMux. Handle and HandleFunc add handlers to DefaultServeMux.

Look at http.Server, alot of fields are optional and works fine with default values.

Now lets look at http.ListenAndServe, is not hard at all

func ListenAndServe(addr string, handler Handler) error {
    server := &Server{Addr: addr, Handler: handler}
    return server.ListenAndServe()
}

so the default server is super simple to create.

func (srv *Server) ListenAndServe() error {
    addr := srv.Addr
    if addr == "" {
        addr = ":http"
    }
    ln, err := net.Listen("tcp", addr)
    if err != nil {
        return err
    }
    return srv.Serve(tcpKeepAliveListener{ln.(*net.TCPListener)})
 }

func (srv *Server) Serve(l net.Listener) error {
    defer l.Close()
    if fn := testHookServerServe; fn != nil {
        fn(srv, l)
    }
    var tempDelay time.Duration // how long to sleep on accept failure
    if err := srv.setupHTTP2(); err != nil {
        return err
    }
    for {
        rw, e := l.Accept()
        if e != nil {
            if ne, ok := e.(net.Error); ok && ne.Temporary() {
                if tempDelay == 0 {
                    tempDelay = 5 * time.Millisecond
                } else {
                    tempDelay *= 2
                }
                if max := 1 * time.Second; tempDelay > max {
                    tempDelay = max
                }
                srv.logf("http: Accept error: %v; retrying in %v", e, tempDelay)
                time.Sleep(tempDelay)
                continue
            }
            return e
        }
        tempDelay = 0
        c := srv.newConn(rw)
        c.setState(c.rwc, StateNew) // before Serve can return
        go c.serve()
    }
}

It Listen on "addr" and Accept every connection, then it spawns a goroutine to handle each connection independently. (HTTP/2.0 is a little bit different, but is the same in general).

If you want to control connections you have 2 options:

1. Create custom server (its 3 lines of code) with server.ConnState callback and control client connections from there. (but they will be Accepted by kernel anyway)

2. Create custom server with your own implementation of net.Listener (like LimitedListener) and control connections from there, this way you will have ultimate power over connections.

Since default http.Server has no way to be stopped the second way is the only way to gracefully terminate listener. You can combine two methods to implement different strategies, well its been done already.