Netty concurrency and "Connection reset by peer"

Question

I've built the following simple server, and I'm stress testing it using ab. If I run ab making 3000 total request (300 concurrent) it works. If I run it again, it shows me:

apr_socket_connect(): Connection reset by peer (54) 

And If after this error I try to make a single request with curl without restarting the server, it works. If I run again ab it shows the same error. It seems that it can't handle too many concurrent connections. Below the code:

public static void main(String[] args) throws Exception {

        ServerBootstrap bootstrap = new ServerBootstrap(
                new NioServerSocketChannelFactory(
                        Executors.newCachedThreadPool(),
                        Executors.newCachedThreadPool()));

        bootstrap.setPipelineFactory(new ChannelPipelineFactory() {
            @Override
            public ChannelPipeline getPipeline() throws Exception {
                return Channels.pipeline(new StringEncoder(), new MyServerHandler());
            }
        });

        bootstrap.bind(new InetSocketAddress(9090));
        System.out.println("Running");
}

Here is the handler:

public class MyServerHandler extends SimpleChannelUpstreamHandler {

    private static AtomicLong request = new AtomicLong();

    @Override
    public void channelConnected(ChannelHandlerContext ctx, ChannelStateEvent e)
            throws Exception {  
        ChannelFuture channelFuture = e.getChannel().write("This is request #" + request.incrementAndGet() + "\n");
        channelFuture.addListener(ChannelFutureListener.CLOSE);
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e)
            throws Exception {
        System.out.println(e.getCause());
        e.getChannel().close();
    }

}

As you see it's very simple, it just shows the total number of requests handled. Any tips?

Thanks

Answer 1

'Connection reset by peer' usually means you have written to a connection that has already been closed by the other end. In other words, an application protocol error. You get the error itself on a subsequent read or write.

Answer 2

I don't immediately see anything wrong, but you could try the following to try to get more information:

1. Override channelClosed and output something so that you're 100% sure that Netty is at least trying to close the channel.
2. Use jvisualvm to have a look at the JVM running your server; you should be able to see the threads and whether they're active or not.
3. Write something to System.out server-side on channelConnected so you know that your connections have made it that far (especially for the 2nd run).
4. When you run ab the second time, is there an error for every connection attempt, or just for some?

What I find odd is that it seems to work the first time, but not thereafter. Keep in mind that this may not be a Netty - or even a JVM - issue, but rather the OS somehow limiting the the connection attempts.

I have done some tests with my own Netty test server, and found that starting large batches concurrent connections will produce an unpredictable outcome (most will connect, some will fail, but always a different ratio). As of yet I haven't figure out why that is (yet), but I suspect it is my OS refusing the connections rather than Netty.