Does Java 6 open a default port for JMX remote connections?

Question

My specific question has to do with JMX as used in JDK 1.6: if I am running a Java process using JRE 1.6 with

com.sun.management.jmxremote 

in the command line, does Java pick a default port for remote JMX connections?

Backstory: I am currently trying to develop a procedure to give to a customer that will enable them to connect to one of our processes via JMX from a remote machine. The goal is to facillitate their remote debugging of a situation occurring on a real-time display console. Because of their service level agreement, they are strongly motivated to capture as much data as possible and, if the situation looks too complicated to fix quickly, to restart the display console and allow it to reconnect to the server-side.

I am aware the I could run jconsole on JDK 1.6 processes and jvisualvm on post-JDK 1.6.7 processes given physical access to the console. However, because of the operational requirements and people problems involved, we are strongly motivated to grab the data that we need remotely and get them up and running again.

EDIT: I am aware of the command line port property

com.sun.management.jmxremote.port=portNum 

The question that I am trying to answer is, if you do not set that property at the command line, does Java pick another port for remote monitoring? If so, how could you determine what it might be?

Answer 1

AFAIK,

Here are the possibilites for connecting a JMX client process (a management application like jconsole, jmxterm, mc4j, jvmstat, jmxmonitor, jps, ...) to a JMX server process (the agent).

The protocol connecting JMX client and JMX server is assumed to be 'Java RMI' (aka 'RMI-JRMP'). This should be the default. One can configure other protocols, in particular 'RMI-IIOP' and 'JMXMP'. Special protocols are possible: the MX4J project for example additionally provides SOAP/HTTP and various serialization protocols over HTTP.

Refer to Sun/Oracle docs for details on configuration.

Also have a look at the file jre/lib/management/management.properties in your JDK distribution.

So, the possibilities:

Case 0: The JVM is started without any particular configuration

Before Java 6: The JVM does not behave as a JMX server. Any program that is run inside the JVM may access the JVM's MBeanServer programmatically and use it to do interesting data exchanges between threads or to do JVM monitoring, but no management from outside the JVM process is possible.

Since Java 6: Even if not explicitely configured, one can access JMX functionality of the JVM locally (from the same machine) as described in "Case 1".

Case 1: The JVM is started with -Dcom.sun.management.jmxremote

The JVM is configured to work as a local (same-machine-only) JMX server.

In this case (and in principle only for Sun/Oracle JVMs) a JMX client can connect to the JMX server through memory-mapped files found in /tmp/hsperfdata_[user]. This is alluded to in the Sun documentation and called "local monitoring" (and also the Attach API). It does not work on FAT filesystems as permissions cannot be set correctly there. See this blog entry.

Sun recommends running jconsole on a machine separate from the JMX server as jconsole apparently is a resource hog, so this "local monitoring" thing is not necessarily a good idea.

Local monitoring is, however, rather secure, only being usable locally and being easily controlled through filesystem permissions.

Case 2: The JMX server is started with -Dcom.sun.management.jmxremote.port=[rmiregistryport]

The JVM is configured to work as a JMX server listening on several TCP ports.

The port specified on the command line will be allocated by the JVM and an RMI registry will be available there. The registry advertises a connector named 'jmxrmi'. It points to a second, randomly allocated TCP port (an 'ephemeral' port) on which the JMX RMI server listens and through which actual data exchange takes place.

Local as described in 'Case 1' is always enabled in 'Case 2'.

The JMX server listens on all interfaces by default, so you can connect to it (and control it) by locally connecting to 127.0.0.1:[rmiregistryport] as well by remotely connecting to [any outside IP address]:[some port] remotely.

This implies that you have to look at the security implications. You can make the JVM listen on 127.0.0.1:[rmiregistryport] only by setting -Dcom.sun.management.jmxremote.local.only=true.

It is rather unfortunate that one cannot specify where the ephemeral port will be allocated - it is always chosen randomly at startup. This may well mean that your firewall needs to become the swiss cheese of the damned! However, there are workarounds. In particular, Apache Tomcat sets the ephemeral JMX RMI server port via its JMX Remote Lifecycle Listener. The code to perform this little magic can be found at org.apache.catalina.mbeans.JmxRemoteLifecycleListener.

If you use this method, you might as well make sure that:

1. The JMX client has to authenticate to the JMX server
2. The TCP exchange between the client and server is encrypted using SSL

How that is done is described in the Sun/Oracle documentation

Other approaches

You can do interesting permutations to avoid having to use the RMI protocol. In particular, you could add a servlet engine (like Jetty) to your process. Then add servlets that translate some HTTP-based exchange internally into direct accesses to the JVM's MBeanServer. You would then be in 'case 0' but still have management capabilities, possibly through an HTML-based interface. The JBoss JMX Console is an example of this.

More off-topic, you could use SNMP directly (something I haven't tried) according to this document.

Show and Tell Time

And now it's time for some code to illustrate a JXM exchange. We take inspiration from a Sunoracle tutorial.

This runs on Unix. We use a JVM that is configured as a JMX server using:

-Dcom.sun.management.jmxremote.port=9001

We use lsof to check what TCP ports it is holding open:

lsof -p <processid> -n | grep TCP

One should see something like this, the registry port and the ephemeral port:

java    1068 user  127u  IPv6 125614246                 TCP *:36828 (LISTEN) java    1068 user  130u  IPv6 125614248                 TCP *:9001  (LISTEN) 

We use tcpdump to inspect the packet exchange between JMX client and JMX server:

tcpdump -l -XX port 36828 or port 9001

We set up a file .java.policy in the home directory to allow the client to actually connect remotely:

grant {     permission java.net.SocketPermission      "<JMX server IP address>:1024-65535", "connect,resolve"; }; 

And then we can run this and see what happens:

package rmi;  import java.rmi.registry.LocateRegistry; import java.rmi.registry.Registry;  import javax.management.remote.rmi.RMIConnection; import javax.management.remote.rmi.RMIServer;  public class Rmi {      public static void main(String args[]) throws Exception {         // We need a Security Manager (not necessarily an RMISecurityManager)         if (System.getSecurityManager() == null) {             System.setSecurityManager(new SecurityManager());         }         //         // Define a registry (this is just about building a local data structure)         //          final int comSunManagementJmxRemotePort = 9001;         Registry registry = LocateRegistry.getRegistry("<JMX server IP address>", comSunManagementJmxRemotePort);         //         // List registry entries. The client connects (using TCP) to the server on the         // 'com.sun.management.jmxremote.port' and queries data to fill the local registry structure.         // Among others, a definition for 'jmxrmi' is obtained.         //         System.out.print("Press enter to list registry entries");         System.in.read();         String[] names = registry.list();         for (String name : names) {             System.out.println("In the registry: " + name);         }         //         // 'Looking up' the entry registered under 'jmxrmi' involves opening and tearing down         // a TCP connection to the 'com.sun.management.jmxremote.port', as well as a TCP         // connection to an ephemeral secondary port chosen at server startup.         // The actual object locally obtained is a "javax.management.remote.rmi.RMIServerImpl_Stub"         // indicating where the ephemeral port is.         // "RMIServerImpl_Stub[UnicastRef [liveRef: [endpoint:[$IP:$EPHEMERAL_PORT](remote),objID:[-62fb4c1c:131a8c709f4:-7fff, -3335792051140327600]]]]"                 //         System.out.print("Press enter to get the 'jmxrmi' stub");         System.in.read();         RMIServer jmxrmiServer = (RMIServer)registry.lookup("jmxrmi");         System.out.println(jmxrmiServer.toString());         //         // Now get a "RMI Connection" to the remote. This involves setting up and tearing         // down a TCP connection to the ephemeral port.          //                 System.out.print("Press enter to get the 'RMIConnection'");         System.in.read();         RMIConnection rcon = jmxrmiServer.newClient(null);         //         // Ask away. This involves setting up and tearing         // down a TCP connection to the ephemeral port.          //         System.out.print("Press enter to get the 'domains'");         System.in.read();         for (String domain : rcon.getDomains(null)) {             System.out.println("Domain: " + domain);         }         //         // Ok, that will do. For serious applications, we better use the higher-level JMX classes         //     }    } 

Answer 2

The documentation suggests that the JMX agent uses a local port -- something unreachable from outside the machine -- unless you specify the following property:

com.sun.management.jmxremote.port=portNum

This is for security reasons, as well as for the reason given by Mr Potato Head. Thus, it looks like Java 6 does not open a default remotely accessible port for JMX.

EDIT: Added after the OP added an answer with more information.

Another option you have is to somehow create a local proxy that listens to all local JMX connections and exports this information. This way, you don't need to have such magic configuration of each JVM instance on the server. Instead the local proxy can connect to all JVMs via JMX and then somehow expose this information remotely. I am not positive exactly how you would implement this, but something like this may be less work than what you otherwise have to do to expose all of your JVMs remotely via JMX.