ActiveMQ:'channel inactive for too long' exceptions stop broker messaging

Question

My system has the following parts:

• ActiveMQ broker exposed on tcp, port 61616
• 3 Grails/Spring wars that live in their own Tomcat servers, they publish and consume messages to the JMS broker
• n times remote client system with a JMS listener component to receive client specific messages, connect to the JMS broker through VPN using a hostname and port 61616

So far, all works fine throughout dev, Test and production environments.

We've just connected a new client system in production and we've noticed that it's logs start to report 'channel was inactive for too long' exceptions and drops the connection. Worrying the overall effect of this one client is that it stops all message consumption on the broker so brings are whole system to a halt.

This client listener (using Spring caching connection factory) appears to connect to the JMS broker ok, process some messages, then 3 mins reports the exception. Turned on DEBUG in ActiveMQ and got loads of output, nothing suggesting a warning or error on the broker around the same time though.

Believe that ActiveMQ has some internal keep alive that should keep the connection even if inactive for longer than the default 30 seconds.

Infrastructure guys have monitored the VPN of this client and confirm it stays up and connected the whole time.

Don't believe it is code or Spring config that is at fault, as we have numerous other instances of the listener in different clients and they all behave themselves fine.

Suppose I have 2 questions really:

1. What is causing 'channel inactive' exceptions?
2. Why does this exception in a single client stop ActiveMQ from working?

EDIT - adding exception stacktrace:

2013-04-24 14:02:06,359 WARN  - Encountered a JMSException - resetting the underlying JMS Connection (org.springframework.jms.connection.CachingConnectionFactory)
javax.jms.JMSException: Channel was inactive for too (>30000) long: jmsserver/xxx.xx.xx.xxx:61616
    at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
    at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1833)
    at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1850)
    at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
    at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
    at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
    at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
    at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
    at org.apache.activemq.transport.InactivityMonitor.onException(InactivityMonitor.java:266)
    at org.apache.activemq.transport.InactivityMonitor$4.run(InactivityMonitor.java:186)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:693)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:719)
    at java.lang.Thread.run(Thread.java:813)
Caused by: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>30000) long: jmsserver/xxx.xx.xx.xxx:61616
    ... 4 more

Answer 1

Have you tried the following:

1. Disable the InactivityMonitor; wireFormat.maxInactivityDuration=0 e.g.

   URL: tcp://localhost:61616?wireFormat.maxInactivityDuration=0

2. If you don't wish to disable, have you tried setting it to a high number e.g.: URL: tcp://localhost:61616?wireFormat.maxInactivityDuration=5000000 (just an example - use your own time in ms)

3. Also, ensure that the jar files are the same version for both client and server.

Hope it helps