Is it possible to wait on both channels and file descriptors at the same time in Go?

Question

I know I can wait on multiple channels using the select {} syntax in Go, and wait on multiple file descriptors using syscall.Select() or similar functions. But is it possible to wait on both channels at once?

For background, I want to have a goroutine that accepts for messages over a channel and forwards them over a socket connection (provided by gozmq), while simultaneously waiting for replies on the socket connection.

Due to the thread safety requirements of the underlying library, the socket can only be accessed in one thread at a time, which is why I was wondering if there is a way to handle this from a single goroutine.

Answer 1

Selecting on both a channel and an file descriptor is not possible because the abstractions are at different levels. Channels are handled by the go runtime and file descriptors by the operating system. What you need is to make a bridge between them and this can be done with a net.Pipe().

Pretty much what you need to do is dedicate one goroutine to epoll()/select() to watch your zmq-sockets and a single "wake up" net.Pipe(). This is your poll server. Another goroutine listens on your read and write channels. When someone sends on the read or write channels, the second goroutine would send on the pipe to wake up the poll server.

This is how the net package in the standard library works. I highly recommend reading it for inspiration (or stealing... the BSD license is very liberal).

Here is a description of pollServer from net itself. You may need to read the code to understand what this is saying, but this section from fd.go should be a good place to start looking.

// A pollServer helps FDs determine when to retry a non-blocking
// read or write after they get EAGAIN.  When an FD needs to wait,
// send the fd on s.cr (for a read) or s.cw (for a write) to pass the
// request to the poll server.  Then receive on fd.cr/fd.cw.
// When the pollServer finds that i/o on FD should be possible
// again, it will send fd on fd.cr/fd.cw to wake any waiting processes.
// This protocol is implemented as s.WaitRead() and s.WaitWrite().
//
// There is one subtlety: when sending on s.cr/s.cw, the
// poll server is probably in a system call, waiting for an fd
// to become ready.  It's not looking at the request channels.
// To resolve this, the poll server waits not just on the FDs it has
// been given but also its own pipe.  After sending on the
// buffered channel s.cr/s.cw, WaitRead/WaitWrite writes a
// byte to the pipe, causing the pollServer's poll system call to
// return.  In response to the pipe being readable, the pollServer
// re-polls its request channels.
//
// Note that the ordering is "send request" and then "wake up server".
// If the operations were reversed, there would be a race: the poll
// server might wake up and look at the request channel, see that it
// was empty, and go back to sleep, all before the requester managed
// to send the request.  Because the send must complete before the wakeup,
// the request channel must be buffered.  A buffer of size 1 is sufficient
// for any request load.  If many processes are trying to submit requests,
// one will succeed, the pollServer will read the request, and then the
// channel will be empty for the next process's request.  A larger buffer
// might help batch requests.
//
// To avoid races in closing, all fd operations are locked and
// refcounted. when netFD.Close() is called, it calls syscall.Shutdown
// and sets a closing flag. Only when the last reference is removed
// will the fd be closed.

Good luck re-implementing net. The good news at the end of all this your zmq-socket will be thread safe in go.

Answer 2

Spawn a new goroutine for each fd you wish to wait on, have them send the fd to a channel when they read something, select on the channels.

Answer 3

All net.Conn's in Go can be accessed concurrently. If your library is based on that, there shouldn't be any problems.

Otherwise, it's quite common in Go to spawn one goroutine per connection. This goroutine often is just responsible for reading from the socket (which might block) and forward all the data to another coordinator goroutine using a channel. This coordinator goroutine is then able to interact with all those socket connections (which are wrapped in a channel) and other events using a select statement. Also note, that you can easily add a buffer to those channels, so that a slow client can't block your coordinator goroutine.