ActiveMQ 5.11.1 Constantly throws EOFException and Won't Send Messages

Question

I have an AMQ 5.11.1 instance running on a few different servers. They are all configured the same and connect to the same gateway. At one particular server, I continually see this message:

2015-08-12 08:53:59,026 | DEBUG | queue://server-01.lookupdata expiring messages .. | org.apache.activemq.broker.region.Queue | ActiveMQ Broker[server-01] Scheduler
2015-08-12 08:53:59,029 | DEBUG | queue://server-01.lookupdata expiring messages done. | org.apache.activemq.broker.region.Queue | ActiveMQ Broker[server-01] Scheduler
2015-08-12 08:54:08,355 | DEBUG | 30000ms elapsed since last read check. | org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ InactivityMonitor ReadCheckTimer
2015-08-12 08:54:08,407 | DEBUG | 30000ms elapsed since last read check. | org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ InactivityMonitor ReadCheckTimer
2015-08-12 08:54:08,411 | INFO  | Network connection between vm://server-01#2 and tcp://gateway/209.221.27.54:55616@34260 shutdown due to a local error: java.io.EOFException | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp://gateway/209.221.27.54:55616@34260
2015-08-12 08:54:08,414 | DEBUG | The local Exception was: java.io.EOFException | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: tcp://gateway/209.221.27.54:55616@34260
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)[:1.7.0_71]
        at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:258)[activemq-client-5.11.1.jar:5.11.1]
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)[activemq-client-5.11.1.jar:5.11.1]
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)[activemq-client-5.11.1.jar:5.11.1]
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)[activemq-client-5.11.1.jar:5.11.1]
        at java.lang.Thread.run(Thread.java:745)[:1.7.0_71]
2015-08-12 08:54:08,418 | DEBUG |  stopping server-01 bridge to production | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ BrokerService[server-01] Task-2
2015-08-12 08:54:08,420 | DEBUG | Unregistering MBean org.apache.activemq:brokerName=server-01,connector=networkConnectors,networkConnectorName=linkToGatewayController1,networkBridge=tcp_//209.221.27.54_55616,type=Broker | org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ BrokerService[server-01] Task-2

The debug output is not helpful, and I can't determine what the difference is between server instances that causes this connection to continually drop.

It has plenty of memory:

jps -v | grep activemq
40359 activemq.jar -Xmx8G

And here's part of my activemq.xml:

   <networkConnector name="linkToGateway"
       uri="static:(tcp://gateway:55616)"
       networkTTL="3"
       duplex="false"
       prefetchSize="2"
   >
        <excludedDestinations>
            <queue physicalName=">"/>
        </excludedDestinations>
        <staticallyIncludedDestinations>
            <queue physicalName="server-01.lookupdata" />
        </staticallyIncludedDestinations>
    </networkConnector>

Any help would be appreciated!

Answer 1

I've had the same problem and solved it by configuring the OpenWire protocol. Default OpenWire settings expect traffic on a connection within 10 seconds and afterwards within a period of 30 seconds. You will see the actually used values in the log. I disabled this timeout (that's ok in my scenario...) by setting: wireFormat.maxInactivityDuration=0

the whole URL:

tcp://localhost:61616?connectionTimeout=0&keepAlive=true&useInactivityMonitor=false&wireFormat.maxInactivityDuration=0

Or make sure that you use the connection immediately and periodically.

http://activemq.apache.org/configuring-wire-formats.html

Answer 2

Originally I was going to mark Tim Bish's answer as correct, but it seems he removed it.

The network connection was the issue. Certain connections were being killed, and after investigating and clearing those issues things started flowing smoothly.