I am trying to remotely monitor a JVM running in docker. The configuration looks like this:
machine 1: runs a JVM (in my case, running kafka) in docker on an ubuntu machine; the IP of this machine is 10.0.1.201; the application running in docker is at 172.17.0.85.
machine 2: runs JMX monitoring
Note that when I run JMX monitoring from machine 2, it fails with a version of the following error (note: the same error occurs when I run jconsole, jvisualvm, jmxtrans, and node-jmx/npm:jmx):
The stack trace upon failing looks something like the following for each of the JMX monitoring tools:
java.rmi.ConnectException: Connection refused to host: 172.17.0.85; nested exception is java.net.ConnectException: Operation timed out at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) (followed by a large stack trace)
Now the interesting part is when I run the same tools (jconsole, jvisualvm, jmxtrans, and node-jmx/npm:jmx) on the same machine that is running docker (machine 1 from above) the JMX monitoring works properly.
I think this suggests that my JMX port is active and working properly, but that when I execute JMX monitoring remotely (from machine 2) it looks like the JMX tool does not recognize the internal docker IP (172.17.0.85)
Below are the relevant (I think) network configuration elements on machine 1 where JMX monitoring works (note the docker ip, 172.17.42.1):
docker0 Link encap:Ethernet HWaddr ... inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0 inet6 addr:... Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:6787941 errors:0 dropped:0 overruns:0 frame:0 TX packets:4875190 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1907319636 (1.9 GB) TX bytes:639691630 (639.6 MB) wlan0 Link encap:Ethernet HWaddr ... inet addr:10.0.1.201 Bcast:10.0.1.255 Mask:255.255.255.0 inet6 addr:... Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4054252 errors:0 dropped:66 overruns:0 frame:0 TX packets:2447230 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2421399498 (2.4 GB) TX bytes:1672522315 (1.6 GB)
And this is the relevant network configuration elements on the remote machine (machine 2) from which I am getting the JMX errors:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384 options=3<RXCSUM,TXCSUM> inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 nd6 options=1<PERFORMNUD> en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500 ether .... inet6 ....%en1 prefixlen 64 scopeid 0x5 inet 10.0.1.203 netmask 0xffffff00 broadcast 10.0.1.255 nd6 options=1<PERFORMNUD> media: autoselect status: active
Remote JMX ConnectionsRight click anywhere in the blank area under the application tree and select Add JMX Connection. Provide the machine name and port number for a running JMX agent, that has been started with the appropriate system properties to allow remote management.
To open the JMX port on the remote JVM, you must enter the port number to use for the JMX RMI connection. Be sure to specify an unused port number. From a command line, go to the bin directory in the <JRE_HOME> directory that contains the Java Runtime Environment (JRE) implementation, for example jre/bin.
The jconsole executable can be found in JDK_HOME/bin, where JDK_HOME is the directory in which the Java Development Kit (JDK) is installed. If this directory is in your system path, you can start JConsole by simply typing jconsole in a command (shell) prompt.
The registry advertises a connector named 'jmxrmi'. It points to a second, randomly allocated TCP port (an 'ephemeral' port) on which the JMX RMI server listens and through which actual data exchange takes place.
For completeness, the following solution worked. The JVM should be run with specific parameters established to enable remote docker JMX monitoring were as followed:
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=<PORT> -Dcom.sun.management.jmxremote.rmi.port=<PORT> -Djava.rmi.server.hostname=<IP> where: <IP> is the IP address of the host that where you executed 'docker run' <PORT> is the port that must be published from docker where the JVM's JMX port is configured (docker run --publish 7203:7203, for example where PORT is 7203). Both `port` and `rmi.port` can be the same.
Once this is done you should be able to execute JMX monitoring (jmxtrans, node-jmx, jconsole, etc) from either a local or remote machine.
Thanks to @Chris-Heald for making this a really quick and simple fix!
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With