Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Tomcat gradually running out of memory with WebSocket apps deployed

I have Tomcat 8.5.9 running on an AWS box with 10 different WebSocket apps deployed that each basically act as a message broker. The https connector is using the Http11NioProtocol. The only parameter I have set is the maxThreads=200 along with the certificate info.

The request volume is not very high. It has been running since Monday morning, and here is what the manager status says:

Max threads: 200
Current thread count: 38
Current thread busy: 0
Keep alive sockets count: 1
Max processing time: 234 ms
Processing time: 17.254 s
Request count: 33351
Error count: 325
Bytes received: 0.00 MB
Bytes sent: 34.07 MB

After a few days, I notice the memory usage continue to grow. I have to restart Tomcat services about every two weeks or so to prevent getting an OutOfMemoryException.

I have been taking heap dumps and analyzing using the Eclipse MAT, which always points to the WsFrameServer class as being the problem suspect. The most recent dump displays the following:

5,146 instances of "org.apache.tomcat.websocket.server.WsFrameServer",
loaded by "java.net.URLClassLoader @ 0x6c0047c28" occupy 1,383,143,200
(73.13%) bytes. These instances are referenced from one instance of
"java.util.concurrent.ConcurrentHashMap$Node[]"

The Dominator Tree is currently has 106,000 entries, most of which are the WsFrameServer class.

Am I doing something wrong or is this "normal"? Are there any specific settings either on Tomcat or on the Connector that I should be setting to prevent this from happening?

Thanks in advance.

EDIT: I'm not sure if this is helpful, but here is what the VisualVM monitor looks like:

VisualVM Monitor

like image 991
Tommo Avatar asked Jul 21 '17 16:07

Tommo


1 Answers

Hard to be certain without more detail but this is probably related to your session retention. What I think is happening is that the WsFrameServer which extends WsFrameBase is added into the session.
If you have an unlimited session retention policy then you will eventually run out of memory.

Try setting a non-0 sessionTimeout

like image 98
Magnus Avatar answered Nov 05 '22 19:11

Magnus