Remote monitoring a Tomcat 7 server using VisualVM and SSH

Question

Sorry if this is a repeat of a question asked previously, but I have tried out the solutions provided here and elsewhere and none of them seem to work for me.

Scenario: I need to profile a production Tomcat 7 server that lives behind a firewall. I only have SSH access to this server. The profiling tool of choice is Visualvm and these servers do not have a GUI installed. Both the server and the machine I am trying to connect from run Ubuntu 10.04 server LTS.

Possible solution: Using visualvm over SSH tunnels seem to be possible and the link given below gives a pretty detailed guide to doing this:

Tomcat+VisualVM+SSH guide

Also I have added a jmxremote.access file to the /conf dir in Tomcat with the following two lines:

monitorRole readonly
controlRole readwrite

Problem: I followed the steps provided in the article and everything seems to be fine. I can connect to the jmx ports in the server locally (from the server) using jmxterm and the connection string given in the guide.

I configured local tomcat as per the guide as a test and Visualvm had no problem connecting to this tomcat instance using the connection string.

However, VisualVM will not connect to the remote server over SSH tunnel. I tried JConsole as it it simpler and has better logging facilities and I get the following sequence of messages:

FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi://localhost:12009/jndi/rmi://localhost:12008/jmxrmi]  connecting...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi://localhost:12009/jndi/rmi://localhost:12008/jmxrmi] finding stub...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi://localhost:12009/jndi/rmi://localhost:12008/jmxrmi] connecting stub...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi://localhost:12009/jndi/rmi://localhost:12008/jmxrmi] getting connection...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi://localhost:12009/jndi/rmi://localhost:12008/jmxrmi] failed to connect: java.rmi.NoSuchObjectException: no such object in table

Any ideas as to what I've done wrong here? Any help is greatly appreciated!

Thank you.

Answer 1

I think the issue you are having, which the referenced article does not really address, is the way that jmx-rmi, or more specifically, RMI itself works. Your Tomcat server setup like this:

1. Created an RMI remotable that is configured to connect to localhost:12009.
2. Created an RMI registry on localhost with a listening port of localhost:12008.

When you run locally, you lookup the stub in the RMI registry, and when you invoke against the stub to issue a jmx call, it connects to localhost:12009, no problem.

When you run remotely through SSH, I am assuming you have tunneled from your remote to the tomcat server on port 12008, so when you issue your request, your remote's localhost:12008 is tunneled to tomcatServer:12008 since the log output seems to indicate that you found the stub ok.

Next, your remote executes against the stub which in turn attempts to connect to localhost:12009 and fails.

My guess is that you need to tunnel your remote's localhost:12008 and localhost:12009 to be redirected to tomcatServer:12008 and tomcatServer:12009.

Another way of getting around this is to ditch the RMI jmx protocol and use something like Jolokia (HTTP), JMX-WS (HTTP WebService) or JMXMP (Socket). This will make your SSH tunelling life a lot easier. If you want to try JMXMP, I created a github project called OpenDMK which has mavenized builds. You will want the jmx-optional package.