VisualVM through firewalls - RMI troubleshooting

Question

Sorry for this question which must have been asked many times, but I can't succeed in resolving my problem. I've read a lot of blogs, sites, forums, .... and didn't find any solution in my case.

Case : I need to connect VisualVM on my box to distant servers (tomcats, weblogics) for performance / threads / memory monitoring. Those servers are installed on (physical or virtual) machines which are protected by a firewall. Large intervals of ports are open in the firewall and can be used, but not all ports.

Tests

• I've tried direct connections through JMX in VisualVM, using following JVM options on the server-side at server startup :

  -Djava.rmi.server.hostname=[hostname]
  -Dcom.sun.management.jmxremote
  -Dcom.sun.management.jmxremote.port=[port]
  -Dcom.sun.management.jmxremote.ssl=false
  -Dcom.sun.management.jmxremote.authenticate=false
  

I've precised the hostname because from my network the hostname and the IP address of the server are not the same than those from the network of the remote server.

No success, VisualVM always seems searching for an unknown server.

• tried starting jstatd on the server-side on a port accessible (-p option) from my box (telnet on this port works), but when launching visualVM on this host with the jstatd port, it still seems waiting for something unreachable.... Same behavior with jps connecting to this remote host.

• tried using the same tools on a server with less network protection, and it works. So I have seen the connections between my box and the server and they are done on ports different from what I've specified to jstatd. I understand that this port is needed for first communication (kind of handshake) and real communications are done on other ports, but not predictible (ex: 60305, 55197, ...). Not sure I understand very well how RMI works.

Please, help me, I'm going crazy !

Answer 1

Unfortunately JMX tries to open ports other than the one you configure. Just yesterday I succeeded connecting to tomcat behind firewall via JMX. The two tricky parts are:

• put a file called jmxremote.access in CATALINA_HOME/conf, which contains the following lines:

  monitorRole readonly
  controlRole readwrite
  

• in server.xml set the ports that will be used by jmx, via a special tomcat listener (catalina-jmx-remote.jar required in /lib):

  <Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" 
      rmiRegistryPortPlatform="9009" rmiServerPortPlatform="9010" />
  

Then open these two ports on the firewall. It works. But that's just for tomcat.

Another option is to use ssh tunnelling. In short - you connect via SSH and configure it to forward some local port (where the jmx client is running) to some ports on the other side of the tunnel.

References:

• Connecting to JMX on Tomcat 6 through a firewall
• Connecting MySQL server on Amazon EC2 instance using ssh tunnel