unable to connect to jmx port remotely

Question

I'm trying to connect to a jmx port remotely but I can't seem to connect to it even though the port is open. Its a java process running in a container on a server thats a Nomad worker. Its running on 29406.

Here is what netstat shows:

netstat -tulpn | grep 29406
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 10.137.2.166:29406      0.0.0.0:*               LISTEN      -               
udp        0      0 10.137.2.166:29406      0.0.0.0:*                           -   

And this is whats in /etc/hosts

cat /etc/hosts
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

I've downloaded jmxterm on the server to try and connect to it, and noticed an interesting behavior. When I try using localhost to connect to the port, I get this:

#RuntimeIOException: Runtime IO exception: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)]

When I use its own IP address, it then seems to work:

$>open 10.137.2.166:29406
#Connection to 10.137.2.166:29406 is opened
$>

Curious to understand why localhost doesn't work when I'm running this on the server itself...

The only way I've gotten jconsole (running on my laptop) to connect to it is by using an ssh tunnel like this:

ssh -Nf -D 7777 10.137.2.166
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=7777 service:jmx:rmi:///jndi/rmi://10.137.2.166:29406/jmxrmi -J-DsocksNonProxyHosts=

I feel like I should be able to connect to it without creating a tunnel but unsure why I can't. If I run telnet locally from my laptop to the host, the connection does seem to open...

telnet 10.137.2.166 29406
Trying 10.137.2.166...
Connected to 10.137.2.166.
Escape character is '^]'.

Answer 1

To successful JMX handshake

1. the jmx server should be available by a host name outside (should also be declared on server jvm via java.rmi.server.hostname system property)

2. in addition to one open port (can be explicitly declared via com.sun.management.jmxremote.rmi.port jvm property) the jmx server chooses random another that's used for new jmx connection. It's quite problematic because you can't foresee particular port in order to exclude it from server's firewall restrictions, so the tunneling is necessary.

Answer 2

Server listened at only 10.137.2.166. When you trying to create new socket with localhost domain, your application tying to establish 127.0.0.1 adress but your application not listening at this ip.

If you want to connect with localhost domain you have few options for solving.

1. Change your server configuration to listen on 127.0.0.1 and 10.137.2.166 at same time.

2. Change your server configuration to listen on 0.0.0.0 .

   Listening at 0.0.0.0 its not recommended for security reasons .

3. Use iptables to forward port. Requires root privileges.

sysctl net.ipv4.ip_forward=1
iptables -t nat -A PREROUTING -p tcp -i lo --dport 29406 -j DNAT --to-destination 10.137.2.166:29406
iptables -A FORWARD -p tcp -d 10.137.2.166 --dport 29406 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

4. if you don't have root privileges you can use socat.

socat TCP-LISTEN:29406,fork,bind=127.0.0.1 TCP:10.137.2.166:29406