JMeter 4.0: Error starting remote server

Question

I am trying to start a test on a Linux slave from a Windows master and get a FileNotFoundException for the rmi_keystore.jks file.

Windows master log file:

2018-05-01 09:13:56,336 INFO o.a.j.e.DistributedRunner: Configuring remote engine: XXX.XXX.XXX.37
2018-05-01 09:13:57,487 INFO o.a.j.e.DistributedRunner: Starting remote engines
2018-05-01 09:13:57,492 INFO o.a.j.e.DistributedRunner: Starting the test @ Tue May 01 09:13:57 CEST 2018 (1525158837487)
2018-05-01 09:13:57,492 INFO o.a.j.e.ClientJMeterEngine: running clientengine run method
2018-05-01 09:13:57,495 INFO o.a.j.r.RmiUtils: System property 'java.rmi.server.hostname' is not defined, using localHost address
2018-05-01 09:13:57,495 INFO o.a.j.r.RmiUtils: Local IP address=XXX.XXX.XXX.38
2018-05-01 09:13:57,498 INFO o.a.j.s.SampleEvent: List of sample_variables: []
2018-05-01 09:13:57,505 INFO o.a.j.r.SSLRMIServerSocketFactory: Created SSLSocket: [SSL: ServerSocket[addr=masterhostname/XXX.XXX.XXX.38,localport=49957]]
2018-05-01 09:13:57,508 INFO o.a.j.r.RmiUtils: System property 'java.rmi.server.hostname' is not defined, using localHost address
2018-05-01 09:13:57,508 INFO o.a.j.r.RmiUtils: Local IP address=XXX.XXX.XXX.38
2018-05-01 09:13:57,511 INFO o.a.j.r.SSLRMIServerSocketFactory: Created SSLSocket: [SSL: ServerSocket[addr=masterhostname/XXX.XXX.XXX.38,localport=49958]]
2018-05-01 09:13:57,542 ERROR o.a.j.e.ClientJMeterEngine: Error in rconfigure() method java.rmi.ConnectIOException: Exception creating connection to: XXX.XXX.XXX.37; nested exception is: 
    java.io.FileNotFoundException: \home\myuser\apache-jmeter-4.0\bin\rmi_keystore.jks (The system cannot find the path specified)
2018-05-01 09:13:57,544 INFO o.a.j.e.ClientJMeterEngine: Interrupting RMI Reaper
2018-05-01 09:14:20,343 INFO o.a.j.e.DistributedRunner: Remote engines have been started

I have created the rmi_keystore.jks file on the master using the create-rmi-keystore.bat script and copied it to apache-jmeter-4.0/bin on the slaves.

After having done so, when trying to start jmeter-server on the slaves, I would get a FileNotFoundException here also (see log and stacktrace below under "Linux slave jmeter-server.log 1").

Setting the property "server.rmi.ssl.keystore.file" to "/home/myuser/apache-jmeter-4.0/bin/rmi_keystore.jks" however fixed this issue so that the slaves now start up with the following prompt output:

[myuser@slave1 ~]$ apache-jmeter-4.0/bin/jmeter-server
Created remote object: UnicastServerRef2 [liveRef: [endpoint:[XXX.XXX.XXX.37:48508,SSLRMIServerSocketFactory(host=slave1.mycompany.corp/XXX.XXX.XXX.37, keyStoreLocation=/home/myuser/apache-jmeter-4.0/bin/rmi_keystore.jks, type=JKS, trustStoreLocation=/home/myuser/apache-jmeter-4.0/bin/rmi_keystore.jks, type=JKS, alias=rmi),SSLRMIClientSocketFactory(keyStoreLocation=/home/myuser/apache-jmeter-4.0/bin/rmi_keystore.jks, type=JKS, trustStoreLocation=/home/myuser/apache-jmeter-4.0/bin/rmi_keystore.jks, type=JKS, alias=rmi)](local),objID:[-5f3a9dbc:1631a887fa1:-7fff, 3570375527221520605]]]

... and log info:

2018-05-01 09:30:50,609 INFO o.a.j.u.JMeterUtils: Setting Locale to en_EN
2018-05-01 09:30:50,639 INFO o.a.j.JMeter: Loading user properties from: /home/myuser/apache-jmeter-4.0/bin/user.properties
2018-05-01 09:30:50,639 INFO o.a.j.JMeter: Loading system properties from: /home/myuser/apache-jmeter-4.0/bin/system.properties
2018-05-01 09:30:50,640 INFO o.a.j.JMeter: Setting System property: server_port=1099
2018-05-01 09:30:50,645 INFO o.a.j.JMeter: Copyright (c) 1998-2018 The Apache Software Foundation
2018-05-01 09:30:50,645 INFO o.a.j.JMeter: Version 4.0 r1823414
2018-05-01 09:30:50,645 INFO o.a.j.JMeter: java.version=1.8.0_172
2018-05-01 09:30:50,646 INFO o.a.j.JMeter: java.vm.name=Java HotSpot(TM) 64-Bit Server VM
2018-05-01 09:30:50,646 INFO o.a.j.JMeter: os.name=Linux
2018-05-01 09:30:50,646 INFO o.a.j.JMeter: os.arch=amd64
2018-05-01 09:30:50,646 INFO o.a.j.JMeter: os.version=2.6.32-573.12.1.el6.x86_64
2018-05-01 09:30:50,646 INFO o.a.j.JMeter: file.encoding=UTF-8
2018-05-01 09:30:50,646 INFO o.a.j.JMeter: Max memory     =1073741824
2018-05-01 09:30:50,646 INFO o.a.j.JMeter: Available Processors =8
2018-05-01 09:30:50,651 INFO o.a.j.JMeter: Default Locale=English (EN)
2018-05-01 09:30:50,651 INFO o.a.j.JMeter: JMeter  Locale=English (EN)
2018-05-01 09:30:50,651 INFO o.a.j.JMeter: JMeterHome=/home/myuser/apache-jmeter-4.0
2018-05-01 09:30:50,652 INFO o.a.j.JMeter: user.dir  =/home/myuser
2018-05-01 09:30:50,652 INFO o.a.j.JMeter: PWD       =/home/myuser
2018-05-01 09:30:50,653 INFO o.a.j.JMeter: IP: XXX.XXX.XXX.37 Name: slave1.mycompany.corp FullName: XXX.XXX.XXX.37
2018-05-01 09:30:50,667 INFO o.a.j.r.RmiUtils: System property 'java.rmi.server.hostname' is not defined, using localHost address
2018-05-01 09:30:50,667 INFO o.a.j.r.RmiUtils: Local IP address=XXX.XXX.XXX.37
2018-05-01 09:30:50,878 INFO o.a.j.r.SSLRMIServerSocketFactory: Created SSLSocket: [SSL: ServerSocket[addr=slave1.mycompany.corp/XXX.XXX.XXX.37,localport=52906]]
2018-05-01 09:30:50,885 INFO o.a.j.e.RemoteJMeterEngineImpl: Starting backing engine on 1099
2018-05-01 09:30:50,885 INFO o.a.j.r.RmiUtils: System property 'java.rmi.server.hostname' is not defined, using localHost address
2018-05-01 09:30:50,886 INFO o.a.j.r.RmiUtils: Local IP address=XXX.XXX.XXX.37
2018-05-01 09:30:50,886 INFO o.a.j.e.RemoteJMeterEngineImpl: IP address is a site-local address; this may cause problems with remote access.
        Can be overridden by defining the system property 'java.rmi.server.hostname' - see jmeter-server script file
2018-05-01 09:30:50,886 INFO o.a.j.e.RemoteJMeterEngineImpl: Creating RMI registry (server.rmi.create=true)
2018-05-01 09:30:50,886 INFO o.a.j.r.RmiUtils: System property 'java.rmi.server.hostname' is not defined, using localHost address
2018-05-01 09:30:50,886 INFO o.a.j.r.RmiUtils: Local IP address=XXX.XXX.XXX.37
2018-05-01 09:30:50,889 INFO o.a.j.r.SSLRMIServerSocketFactory: Created SSLSocket: [SSL: ServerSocket[addr=slave1.mycompany.corp/XXX.XXX.XXX.37,localport=1099]]
2018-05-01 09:30:50,890 INFO o.a.j.e.RemoteJMeterEngineImpl: Bound to RMI registry on port 1099

Can anyone tell what is wrong with the setup? Is there a problem with controlling Linux slaves from a Windows master (notice the backslashes in the FileNotFoundException in the master log)?

I should perhaps mention that I have set remote_hosts=XXX.XXX.XXX.36,XXX.XXX.XXX.37 in the master jmeter.properties file and with success have tried to create a telnet connection from the master to the slaves on port 1099.

Linux slave jmeter-server.log 1:

2018-05-01 09:24:14,970 INFO o.a.j.u.JMeterUtils: Setting Locale to en_EN
2018-05-01 09:24:15,000 INFO o.a.j.JMeter: Loading user properties from: /home/myuser/apache-jmeter-4.0/bin/user.properties
2018-05-01 09:24:15,000 INFO o.a.j.JMeter: Loading system properties from: /home/myuser/apache-jmeter-4.0/bin/system.properties
2018-05-01 09:24:15,001 INFO o.a.j.JMeter: Setting System property: server_port=1099
2018-05-01 09:24:15,006 INFO o.a.j.JMeter: Copyright (c) 1998-2018 The Apache Software Foundation
2018-05-01 09:24:15,006 INFO o.a.j.JMeter: Version 4.0 r1823414
2018-05-01 09:24:15,006 INFO o.a.j.JMeter: java.version=1.8.0_172
2018-05-01 09:24:15,006 INFO o.a.j.JMeter: java.vm.name=Java HotSpot(TM) 64-Bit Server VM
2018-05-01 09:24:15,007 INFO o.a.j.JMeter: os.name=Linux
2018-05-01 09:24:15,007 INFO o.a.j.JMeter: os.arch=amd64
2018-05-01 09:24:15,007 INFO o.a.j.JMeter: os.version=2.6.32-573.12.1.el6.x86_64
2018-05-01 09:24:15,007 INFO o.a.j.JMeter: file.encoding=UTF-8
2018-05-01 09:24:15,007 INFO o.a.j.JMeter: Max memory     =1073741824
2018-05-01 09:24:15,007 INFO o.a.j.JMeter: Available Processors =8
2018-05-01 09:24:15,012 INFO o.a.j.JMeter: Default Locale=English (EN)
2018-05-01 09:24:15,012 INFO o.a.j.JMeter: JMeter  Locale=English (EN)
2018-05-01 09:24:15,012 INFO o.a.j.JMeter: JMeterHome=/home/myuser/apache-jmeter-4.0
2018-05-01 09:24:15,012 INFO o.a.j.JMeter: user.dir  =/home/myuser
2018-05-01 09:24:15,013 INFO o.a.j.JMeter: PWD       =/home/myuser
2018-05-01 09:24:15,014 INFO o.a.j.JMeter: IP: XXX.XXX.XXX.37 Name: slave1.mycompany.corp FullName: XXX.XXX.XXX.37
2018-05-01 09:24:15,027 INFO o.a.j.r.RmiUtils: System property 'java.rmi.server.hostname' is not defined, using localHost address
2018-05-01 09:24:15,027 INFO o.a.j.r.RmiUtils: Local IP address=XXX.XXX.XXX.37
2018-05-01 09:24:15,041 ERROR o.a.j.JMeter: Giving up, as server failed with:
java.rmi.server.ExportException: Listen failed on port: 0; nested exception is:
        java.io.FileNotFoundException: rmi_keystore.jks (No such file or directory)
        at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:348) ~[?:1.8.0_172]
        at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:254) ~[?:1.8.0_172]
        at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411) ~[?:1.8.0_172]
        at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147) ~[?:1.8.0_172]
        at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:236) ~[?:1.8.0_172]
        at java.rmi.server.UnicastRemoteObject.exportObject(UnicastRemoteObject.java:383) ~[?:1.8.0_172]
        at java.rmi.server.UnicastRemoteObject.exportObject(UnicastRemoteObject.java:346) ~[?:1.8.0_172]
        at java.rmi.server.UnicastRemoteObject.<init>(UnicastRemoteObject.java:225) ~[?:1.8.0_172]
        at org.apache.jmeter.engine.RemoteJMeterEngineImpl.<init>(RemoteJMeterEngineImpl.java:65) ~[ApacheJMeter_core.jar:4.0 r1823414]
        at org.apache.jmeter.engine.RemoteJMeterEngineImpl.startServer(RemoteJMeterEngineImpl.java:71) ~[ApacheJMeter_core.jar:4.0 r1823414]
        at org.apache.jmeter.JMeter.start(JMeter.java:501) [ApacheJMeter_core.jar:4.0 r1823414]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_172]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_172]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_172]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172]
        at org.apache.jmeter.NewDriver.main(NewDriver.java:245) [ApacheJMeter.jar:4.0 r1823414]
Caused by: java.io.FileNotFoundException: rmi_keystore.jks (No such file or directory)
        at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_172]
        at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_172]
        at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[?:1.8.0_172]
        at java.io.FileInputStream.<init>(FileInputStream.java:93) ~[?:1.8.0_172]
        at org.apache.jmeter.rmi.SSLRMIServerSocketFactory.loadStore(SSLRMIServerSocketFactory.java:148) ~[ApacheJMeter_core.jar:4.0 r1823414]
        at org.apache.jmeter.rmi.SSLRMIServerSocketFactory.createServerSocket(SSLRMIServerSocketFactory.java:100) ~[ApacheJMeter_core.jar:4.0 r1823414]
        at sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666) ~[?:1.8.0_172]
        at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335) ~[?:1.8.0_172]
        ... 15 more
2018-05-01 09:24:15,046 ERROR o.a.j.JMeter: An error occurred:
java.rmi.server.ExportException: Listen failed on port: 0; nested exception is:
        java.io.FileNotFoundException: rmi_keystore.jks (No such file or directory)
        at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:348) ~[?:1.8.0_172]
        at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:254) ~[?:1.8.0_172]
        at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411) ~[?:1.8.0_172]
        at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147) ~[?:1.8.0_172]
        at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:236) ~[?:1.8.0_172]
        at java.rmi.server.UnicastRemoteObject.exportObject(UnicastRemoteObject.java:383) ~[?:1.8.0_172]
        at java.rmi.server.UnicastRemoteObject.exportObject(UnicastRemoteObject.java:346) ~[?:1.8.0_172]
        at java.rmi.server.UnicastRemoteObject.<init>(UnicastRemoteObject.java:225) ~[?:1.8.0_172]
        at org.apache.jmeter.engine.RemoteJMeterEngineImpl.<init>(RemoteJMeterEngineImpl.java:65) ~[ApacheJMeter_core.jar:4.0 r1823414]
        at org.apache.jmeter.engine.RemoteJMeterEngineImpl.startServer(RemoteJMeterEngineImpl.java:71) ~[ApacheJMeter_core.jar:4.0 r1823414]
        at org.apache.jmeter.JMeter.start(JMeter.java:501) [ApacheJMeter_core.jar:4.0 r1823414]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_172]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_172]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_172]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172]
        at org.apache.jmeter.NewDriver.main(NewDriver.java:245) [ApacheJMeter.jar:4.0 r1823414]
Caused by: java.io.FileNotFoundException: rmi_keystore.jks (No such file or directory)
        at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_172]
        at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_172]
        at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[?:1.8.0_172]
        at java.io.FileInputStream.<init>(FileInputStream.java:93) ~[?:1.8.0_172]
        at org.apache.jmeter.rmi.SSLRMIServerSocketFactory.loadStore(SSLRMIServerSocketFactory.java:148) ~[ApacheJMeter_core.jar:4.0 r1823414]
        at org.apache.jmeter.rmi.SSLRMIServerSocketFactory.createServerSocket(SSLRMIServerSocketFactory.java:100) ~[ApacheJMeter_core.jar:4.0 r1823414]
        at sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666) ~[?:1.8.0_172]
        at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335) ~[?:1.8.0_172]
        ... 15 more

Answer 1

You need to create the keystore prior to starting JMeter slaves using create-rmi-keystore.sh script which is located in "bin" folder of your JMeter installation. See Setting up SSL chapter of the JMeter User Manual on Distributed Testing for more details.

If you want to revert JMeter to previous behaviour (as it used to be prior to JMeter 4.0) you can start JMeter master and slaves like:

jmeter -Jserver.rmi.ssl.disable=true

To make the change permanent add the next line to user.properties file:

server.rmi.ssl.disable=true

and restart JMeter to pick the change up. See Configuring JMeter user manual chapter for more information on tuning JMeter using various properties.