ActiveMQ exception when stopping

Question

Configuration: Redhat 5.3, Sun Java 1.6.31, ActiveMQ 5.4.2

I'm getting an exception every time I stop activemq ($> service activemq stop)

ACTIVEMQ_HOME: /opt/apache/apache-activemq-5.4.2
ACTIVEMQ_BASE: /opt/apache/apache-activemq-5.4.2
Connecting to JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
ERROR: java.lang.RuntimeException: Failed to execute stop task. Reason: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
    java.net.ConnectException: Connection refused]
java.lang.RuntimeException: Failed to execute stop task. Reason: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
    java.net.ConnectException: Connection refused]
    at org.apache.activemq.console.command.ShutdownCommand.runTask(ShutdownCommand.java:107)
    at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
    at org.apache.activemq.console.command.AbstractJmxCommand.execute(AbstractJmxCommand.java:316)
    at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:143)
    at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
    at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:85)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.activemq.console.Main.runTaskClass(Main.java:251)
    at org.apache.activemq.console.Main.main(Main.java:107)
ERROR: java.lang.Exception: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
    java.net.ConnectException: Connection refused]
java.lang.Exception: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
    java.net.ConnectException: Connection refused]
    at org.apache.activemq.console.command.ShutdownCommand.runTask(ShutdownCommand.java:108)
    at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
    at org.apache.activemq.console.command.AbstractJmxCommand.execute(AbstractJmxCommand.java:316)
    at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:143)
    at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
    at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:85)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.activemq.console.Main.runTaskClass(Main.java:251)
    at org.apache.activemq.console.Main.main(Main.java:107)
Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
    java.net.ConnectException: Connection refused]
    at javax.management.remote.rmi.RMIConnector.connect(Unknown Source)
    at javax.management.remote.JMXConnectorFactory.connect(Unknown Source)
    at javax.management.remote.JMXConnectorFactory.connect(Unknown Source)
    at org.apache.activemq.console.command.AbstractJmxCommand.createJmxConnector(AbstractJmxCommand.java:214)
    at org.apache.activemq.console.command.AbstractJmxCommand.createJmxConnection(AbstractJmxCommand.java:237)
    at org.apache.activemq.console.command.ShutdownCommand.runTask(ShutdownCommand.java:71)
    ... 11 more
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
    java.net.ConnectException: Connection refused]
    at com.sun.jndi.rmi.registry.RegistryContext.lookup(Unknown Source)
    at com.sun.jndi.toolkit.url.GenericURLContext.lookup(Unknown Source)
    at javax.naming.InitialContext.lookup(Unknown Source)
    at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(Unknown Source)
    at javax.management.remote.rmi.RMIConnector.findRMIServer(Unknown Source)
    ... 17 more
Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
    java.net.ConnectException: Connection refused
    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(Unknown Source)
    at sun.rmi.transport.tcp.TCPChannel.createConnection(Unknown Source)
    at sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown Source)
    at sun.rmi.server.UnicastRef.newCall(Unknown Source)
    at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
    ... 22 more
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(Unknown Source)
    at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.<init>(Unknown Source)
    at java.net.Socket.<init>(Unknown Source)
    at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(Unknown Source)
    at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(Unknown Source)
    ... 27 more

The process still dies but not gracefully. Any suggestions?

Answer 1

I've found a fix, but I'm not sure it addresses the root problem. Once ActiveMQ has started I checked to see if the port 1099 is actually open using the following command:

$> netstat -lptun | grep 1099

This command prints nothing showing that the port is not open (with ActiveMQ started). This means that even though ActiveMQ has started, it's not listening for any JMX connections (which is how the script will attempt to stop it). After some looking around I found this link that led me to the {ACTIVEMQ_HOME}/conf/activemq.xml file. In this file there is the following section:

 <!-- 
     The managementContext is used to configure how ActiveMQ is exposed in 
     JMX. By default, ActiveMQ uses the MBean server that is started by 
     the JVM. For more information, see: 

         http://activemq.apache.org/jmx.html 
 -->
 <managementContext>
   <managementContext createConnector="false"/>
 </managementContext>

Setting createConnector="true" resolved my issues (this will start the MBean server within the JVM when ActiveMQ starts). The shutdown output now looks like this (as expected):

...
ACTIVEMQ_HOME: /opt/apache/apache-activemq-5.4.2
ACTIVEMQ_BASE: /opt/apache/apache-activemq-5.4.2
Connecting to JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
Stopping broker: localhost
.. FINISHED

Additionally, after changing this flag I can verify that the default JMX port (1099) is open after starting ActiveMQ

$> netstat -lptun | grep 1099
tcp        0      0 :::1099                     :::*                        LISTEN      30033/java