Stop accepting new TCP connections without dropping any existing ones

Question

I have two servers listening on a TCP port behind a load balancer. The load balancer can detect if a TCP connection attempt from a client was unsuccessful and retry it to the second server without dropping that connection. I want to be able to bring any of these two servers down for maintenance without dropping a single client collection.

My servers use this code to process client requests:

ServerSocketFactory ssf = ...
ServerSocket serverSocket = ssf.createServerSocket(60000);
try {
    while (true) {
        Socket socket = serverSocket.accept();
        ...// Do the processing
    }
} catch (IOException e) {
    ...
}
...

My initial thought was to add a boolean that would be set on application shutdown and prevent new serverSocket.accept() calls while waiting for all existing connection to be processed and closed. However, new connection are being established even before the serverSocket.accept() call. Here's what I see in Wireshark if I put a breakpoint before that call. The problem is at this point as soon as I call serverSocket.close(), all such client connections get dropped. What I want to achieve is some way of telling ServerSocket to stop accept all new connections (i.e. only send RST for new connections or let them time out), so the load balancer can reroute them to another server, but at the same time not drop any already established connections.

Edit: I'm looking for some automated solution which wouldn't require me to change any load balancer or OS settings every time I want to update the application.

Answer 1

You could add firewall rule on the server which will block new but keep old connections active. I guess the server is Linux based? If so, you could try with:

iptables -A INPUT -p tcp --syn --destination-port <port> -j REJECT --reject-with icmp-host-prohibited

After that you can check with netstat is there any active connection and bring the application down once when there is no any:

netstat -ant|grep <port>|grep EST

After you finish with the maintenance, you can remove the firewall rule. First, list all the rules to find it:

iptables -L -n

And remove it:

iptables -D INPUT <rule number>