Java Multithreaded Socket Server hangs after getting ~ 50 simultaneous Connections

Question

So basically the problem is described in the title. The server works in the following way:

• Listens to a new connection
• Once connection is requested - adds the request to the Q,
• Continues listening to a new connection
• Separate process takes care of a Q and spawns a new thread to deal with the clients' requests.

The server code is similar to this tutorial (everything is in try / catch, unfortunately I cant show the source-code - company policy)

It seems to work very well, until the number of clients exceeds ~ 50, Then it just hangs with no exceptions / warnings / etc. There is a cpu thread limit of 32k, no limits on the number of open files / open sockets / etc. OS = CentOS 5.5 (same seems to happen in ubuntu tho). The server logs data to MySQL using ODBC. Separate stress tests of both showed that I can have up to 32k java processes (limited by /proc/sys/kernel/threads-max ) and MySQL can perform up to 20k simple operations / second, so Im assuming the problem is with the sockets.

So the question really is:

• What is the limiting factor in socket connections and how can I make it bigger?
• OR am I looking in the wrong place?

Answer 1

The chances are that you have induced a deadlock somewhere in the code. The key indicator here is if by 'hang' you mean the CPU usage of the server drops to nothing and no futher activity is seen in the server.

When the server hangs run jdk tool: jstack against it's process. This should show you what is waiting on what lock. Also in the tool kit is jvisualvm and if on a unix box a simple kill -3 pid will do a thread dump to stderr.

With out the code or at least a reproducable sample I'm afraid I can't help much more. One thing you might want to look at is using jetty as your embedded server instead of a hand roled one, they have already been through the deadlock/threading pain so you don't have to.