Java NIO SocketChannel.read() with multithread

Question

I am implementing a simple file server using Java NIO with one selecting thread and multiple worker threads(for performing real read/write).

The main part of the code looks like the following:

while (true) {
    int num = selector.select();
    if (num > 0) { 
        Iterator<SelectionKey> keys = selector.selectedKeys().iterator();
        final SelectionKey key = keys.next();
        keys.remove();

        if (key.isValid()) {
            if (key.isAcceptable()) {
                accept(key);
            } else if (key.isReadable()) {
                performReadInWorkerThread (key);
            } else if (key.isWritable()) {
                performWriteInWorkerThread (key);
            }
        }
    }
}

As you can see from the code snippet, when a readable/writable channel is selected, I offload the read/write from the selecting thread to a worker thread.

Now the problem is that when a readble channel is handed over to the worker thread, and before it finishes/starts reading from the channel, the selecting thread loops again, and selector.select() selects the previously selected readable channel(because there's still input buffer in the channel that is not yet fully consumed by the previously assigned worker thread), so again the channel is handed over to another worker thread, resulting in multiple worker threads reading the same channel.

I believe this is a design problem. My question is how I can ensure only one thread reading a channel at the same time?

Answer 1

Why? The read won't block. Do it in the current thread. You are just in for endless problems this way. You will have to deregister OP_READ before you hand over to the read thread, which is easy enough, but the hard part is that when the read thread completes the read it will have to re-register OP_READ, which requires either (i) a selector wakeup(), which causes the select thread to run when there is possibly nothing to do, which is wasteful, or else (ii) use a queue of pending reregistrations, which delays the next read on that channel until after the next time the selector wakes up, which is also wasteful, or else you have to wakeup the selector immediately on adding to the queue, which is also wasteful if nothing is ready. I've never seen a convincing NIO architecture that used different select and read threads.

Don't do this. If you must have multithreading, organize your channels into groups, each with its own selector and its own thread, and have all those threads do their own reading.

Similarly there is no need to write in a separate thread. Just write when you have something to write.

Answer 2

For NIO, just keep one principle in mind: Do read/write in the main select thread. This principle is the reflection of the hardware nature. Don't worry that read in main select thread is not swift. In modern server, CPU is always faster than network card. So no-blocking read in one thread is also faster than network card operations. One thread is already enough to read packet. We needn't any more threads.