Shutting down Netty server when client connections are open

Question

I'm trying to shutdown a Netty server that has open connections to it, and it just hangs. Here's what I do.

• Start server on one machine and client on another.
• Send a message from client to server to which I get a response.
• Shutdown server using Ctrl-C

I've registered a shutdown hook on the server that closes the ChannelGroup and calls releaseExternalResources on the ServerBootstrap (or actually I'm using the DuplexTcpServerBootstrap of the protobuf-pro-duplex library that does just that). Anyway, the shutdown hook is called properly on shutdown, but it never returns. When I take a thread dump of what's happening I can see two interesting stacks:

   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000006b0890950> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
    at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1433)
    at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:103)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.releaseExternalResources(AbstractNioWorkerPool.java:80)
    at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.releaseExternalResources(NioServerSocketChannelFactory.java:162)
    at org.jboss.netty.bootstrap.Bootstrap.releaseExternalResources(Bootstrap.java:319)
    at com.googlecode.protobuf.pro.duplex.server.DuplexTcpServerBootstrap.releaseExternalResources(DuplexTcpServerBootstrap.java:132)
    at com.xxx.yyy.node.NodeServer$2.run(NodeServer.java:104)
    at java.lang.Thread.run(Thread.java:722)

So this is the shutdown hook thread that never returns. And below is another thread that seems to be waiting on a channel:

   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
    at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:274)
    at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
    - locked <0x00000006b0896660> (a java.lang.Object)
    at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
    at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:80)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    - locked <0x00000006b08964a8> (a sun.nio.ch.Util$2)
    - locked <0x00000006b0896498> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000006b0890d20> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:208)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

I'm using Netty 3.4.6.Final with Java 7.04 on Linux. Thanks!

Br, Frank.

Answer 1

Had the same 'problem' with bare Netty client/server as well.

Thing is, closing the server channel DOES NOT close the open channels created for accepted client connections. One has to explicitly keep track of client channels in the server. This can be done with channel groups and a handler which adds client channels to this group. When it comes to shutting down the server, all channels in the group should be closed in a batch-like manner instead of just the one server channel (which can put into the channel group as well).

There is excellent documentation in the user guide (9. Shutting Down Your Application): http://static.netty.io/3.5/guide/ and the ChannelGroup API doc (Simplify shutdown process with ChannelGroup): http://static.netty.io/3.5/api/org/jboss/netty/channel/group/ChannelGroup.html