Glassfish 5 will not start after enable-secure-admin command

Question

I can not run Glassfish 5 after entering enable-secure-admin command. I have clear OS Ubutntu server 16.04, then I install java 8 and download glassfish server on zip file. What could be the problem? Can someone faced with a similar problem.

My steps:

Unzip archive

glassfish@ubnt:~$ cd /home/glassfish
glassfish@ubnt:~$ unzip glassfish-5.0.zip

Sart domain1

glassfish@ubnt:~$ glassfish5/bin/asadmin start-domain
Waiting for domain1 to start ...........
Successfully started the domain : domain1
domain  Location: /home/glassfish/glassfish5/glassfish/domains/domain1
Log File: /home/glassfish/glassfish5/glassfish/domains/domain1/logs/server.log
Admin Port: 4848
Command start-domain executed successfully.
Change admin password

glassfish@ubnt:~$ glassfish5/bin/asadmin change-admin-password
Enter admin user name [default: admin]>
Enter the admin password>
Enter the new admin password>
Enter the new admin password again>
Command change-admin-password executed successfully.

Enable secure admin. When I try to restart the domain, I get an error NCLS-ADMIN-00010.

glassfish@ubnt:~$ glassfish5/bin/asadmin enable-secure-admin
You must restart all running servers for the change in secure admin to take effect.
Command enable-secure-admin executed successfully.

glassfish@ubnt:~$ glassfish5/bin/asadmin restart-domain
NCLS-ADMIN-00010
NCLS-ADMIN-00010

glassfish@ubnt:~$ glassfish5/bin/asadmin start-domain
Waiting for domain1 to start .......
Successfully started the domain : domain1
domain  Location: /home/glassfish/glassfish5/glassfish/domains/domain1
Log File: /home/glassfish/glassfish5/glassfish/domains/domain1/logs/server.log
Admin Port: 4848
Command start-domain executed successfully.

glassfish@ubnt:~$ glassfish5/bin/asadmin list-domains
NCLS-ADMIN-00010
domain1 not running
Command list-domains executed successfully.

glassfish@ubnt:~$ glassfish5/bin/asadmin disable-secure-admin
NCLS-ADMIN-00010
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
Command disable-secure-admin failed.

Server log:

[2018-04-23T15:15:13.701+0300] [glassfish 5.0] [INFO] [] [javax.enterprise.system.core] [tid: _ThreadID=132 _ThreadName=Thread-33] [timeMillis: 1524485713701] [levelValue: 800] [[
  Server restart initiated]]

[2018-04-23T15:15:13.804+0300] [glassfish 5.0] [INFO] [NCLS-BOOTSTRAP-00028] [javax.enterprise.bootstrap] [tid: _ThreadID=132 _ThreadName=Thread-33] [timeMillis: 1524485713804] [levelValue: 800] [[
  Unregistered com.sun.enterprise.glassfish.bootstrap.osgi.EmbeddedOSGiGlassFishImpl@7e7e962d from service registry.]]

[2018-04-23T15:15:13.807+0300] [glassfish 5.0] [INFO] [] [] [tid: _ThreadID=132 _ThreadName=Thread-8] [timeMillis: 1524485713807] [levelValue: 800] [[
  FileMonitoring shutdown]]

[2018-04-23T15:15:13.818+0300] [glassfish 5.0] [INFO] [NCLS-JMX-00002] [javax.enterprise.system.jmx] [tid: _ThreadID=132 _ThreadName=Thread-33] [timeMillis: 1524485713818] [levelValue: 800] [[
  JMXStartupService: Stopped JMXConnectorServer: null]]

[2018-04-23T15:15:13.820+0300] [glassfish 5.0] [INFO] [NCLS-JMX-00001] [javax.enterprise.system.jmx] [tid: _ThreadID=132 _ThreadName=Thread-33] [timeMillis: 1524485713820] [levelValue: 800] [[
  JMXStartupService and JMXConnectors have been shut down.]]

[2018-04-23T15:15:14.123+0300] [glassfish 5.0] [INFO] [] [] [tid: _ThreadID=145 _ThreadName=Thread-8] [timeMillis: 1524485714123] [levelValue: 800] [[
  JdbcRuntimeExtension,  getAllSystemRAResourcesAndPools = [GlassFishConfigBean.org.glassfish.jdbc.config.JdbcResource, GlassFishConfigBean.org.glassfish.jdbc.config.JdbcResource, GlassFishConfigBean.org.glassfish.jdbc.config.JdbcConnectionPool, GlassFishConfigBean.org.glassfish.jdbc.config.JdbcConnectionPool]]]

[2018-04-23T15:15:15.488+0300] [] [INFO] [NCLS-GFLAUNCHER-00005] [javax.enterprise.launcher] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1524485715488] [levelValue: 800] [[
  JVM invocation command line:
/usr/lib/jvm/java-8-oracle/bin/java
-cp
/home/glassfish/glassfish5/glassfish/modules/glassfish.jar
-XX:+UnlockDiagnosticVMOptions
-XX:NewRatio=2
-XX:MaxPermSize=192m
-Xmx512m
-javaagent:/home/glassfish/glassfish5/glassfish/lib/monitor/flashlight-agent.jar
-client
-Djavax.xml.accessExternalSchema=all
-Djavax.net.ssl.trustStore=/home/glassfish/glassfish5/glassfish/domains/domain1/config/cacerts.jks
-Djdk.tls.rejectClientInitiatedRenegotiation=true
-Djdk.corba.allowOutputStreamSubclass=true
-Dfelix.fileinstall.dir=/home/glassfish/glassfish5/glassfish/modules/autostart/
-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.shell.remote,org.apache.felix.fileinstall
-Dcom.sun.aas.installRoot=/home/glassfish/glassfish5/glassfish
-Dfelix.fileinstall.poll=5000
-Djava.endorsed.dirs=/home/glassfish/glassfish5/glassfish/modules/endorsed:/home/glassfish/glassfish5/glassfish/lib/endorsed
-Djava.security.policy=/home/glassfish/glassfish5/glassfish/domains/domain1/config/server.policy
-Dosgi.shell.telnet.maxconn=1
-Dfelix.fileinstall.bundles.startTransient=true
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dfelix.fileinstall.log.level=2
-Djavax.net.ssl.keyStore=/home/glassfish/glassfish5/glassfish/domains/domain1/config/keystore.jks
-Djava.security.auth.login.config=/home/glassfish/glassfish5/glassfish/domains/domain1/config/login.conf
-Dfelix.fileinstall.disableConfigSave=false
-Dfelix.fileinstall.bundles.new.start=true
-Dcom.sun.aas.instanceRoot=/home/glassfish/glassfish5/glassfish/domains/domain1
-Dosgi.shell.telnet.port=6666
-Dgosh.args=--nointeractive
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-Dosgi.shell.telnet.ip=127.0.0.1
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-Djava.awt.headless=true
-Dcom.ctc.wstx.returnNullForDefaultNamespace=true
-Djava.ext.dirs=/usr/lib/jvm/java-8-oracle/lib/ext:/usr/lib/jvm/java-8-oracle/jre/lib/ext:/home/glassfish/glassfish5/glassfish/domains/domain1/lib/ext
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-Djava.library.path=/home/glassfish/glassfish5/glassfish/lib:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
com.sun.enterprise.glassfish.bootstrap.ASMain
-upgrade
false
-domaindir
/home/glassfish/glassfish5/glassfish/domains/domain1
-read-stdin
true
-asadmin-args
--host,,,localhost,,,--port,,,4848,,,--secure=false,,,--terse=false,,,--echo=false,,,--interactive=true,,,start-domain,,,--verbose=false,,,--watchdog=false,,,--debug=false,,,--domaindir,,,/home/glassfish/glassfish5/glassfish/domains,,,domain1
-domainname
domain1
-instancename
server
-type
DAS
-verbose
false
-asadmin-classpath
/home/glassfish/glassfish5/glassfish/lib/client/appserver-cli.jar
-debug
false
-asadmin-classname
com.sun.enterprise.admin.cli.AdminMain]]

[2018-04-23T15:15:20.859+0300] [glassfish 5.0] [INFO] [NCLS-LOGGING-00009] [javax.enterprise.logging] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1524485720489] [timeMillis: 1524485720859] [levelValue: 800] [[
  Running GlassFish Version: GlassFish Server Open Source Edition  5.0  (build 25)]]

[2018-04-23T15:15:20.865+0300] [glassfish 5.0] [INFO] [NCLS-LOGGING-00010] [javax.enterprise.logging] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1524485720489] [timeMillis: 1524485720865] [levelValue: 800] [[
  Server log file is using Formatter class: com.sun.enterprise.server.logging.ODLLogFormatter]]

[2018-04-23T15:15:21.185+0300] [glassfish 5.0] [INFO] [NCLS-SECURITY-01115] [javax.enterprise.system.core.security] [tid: _ThreadID=16 _ThreadName=RunLevelControllerThread-1524485720508] [timeMillis: 1524485721185] [levelValue: 800] [[
  Realm [admin-realm] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created.]]

[2018-04-23T15:15:21.191+0300] [glassfish 5.0] [INFO] [NCLS-SECURITY-01115] [javax.enterprise.system.core.security] [tid: _ThreadID=16 _ThreadName=RunLevelControllerThread-1524485720508] [timeMillis: 1524485721191] [levelValue: 800] [[
  Realm [file] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created.]]

[2018-04-23T15:15:21.204+0300] [glassfish 5.0] [INFO] [NCLS-SECURITY-01115] [javax.enterprise.system.core.security] [tid: _ThreadID=16 _ThreadName=RunLevelControllerThread-1524485720508] [timeMillis: 1524485721204] [levelValue: 800] [[
  Realm [certificate] of classtype [com.sun.enterprise.security.auth.realm.certificate.CertificateRealm] successfully created.]]

[2018-04-23T15:15:21.909+0300] [glassfish 5.0] [INFO] [SEC-SVCS-00100] [javax.enterprise.security.services] [tid: _ThreadID=14 _ThreadName=RunLevelControllerThread-1524485720480] [timeMillis: 1524485721909] [levelValue: 800] [[
  Authorization Service has successfully initialized.]]

[2018-04-23T15:15:22.106+0300] [glassfish 5.0] [INFO] [] [org.glassfish.ha.store.spi.BackingStoreFactoryRegistry] [tid: _ThreadID=14 _ThreadName=RunLevelControllerThread-1524485720480] [timeMillis: 1524485722106] [levelValue: 800] [[
  Registered org.glassfish.ha.store.adapter.cache.ShoalBackingStoreProxy for persistence-type = replicated in BackingStoreFactoryRegistry]]

[2018-04-23T15:15:22.332+0300] [glassfish 5.0] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1524485720489] [timeMillis: 1524485722332] [levelValue: 800] [[
  Grizzly Framework 2.4.1 started in: 38ms - bound to [/0.0.0.0:8080]]]

[2018-04-23T15:15:22.394+0300] [glassfish 5.0] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1524485720489] [timeMillis: 1524485722394] [levelValue: 800] [[
  Grizzly Framework 2.4.1 started in: 4ms - bound to [/0.0.0.0:8181]]]

[2018-04-23T15:15:22.427+0300] [glassfish 5.0] [WARNING] [] [org.glassfish.grizzly.config.GenericGrizzlyListener] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1524485720489] [timeMillis: 1524485722427] [levelValue: 900] [[
  HTTP/2 (enabled by default) is unsupported with port unification and will be disabled for network listener admin-listener.]]

[2018-04-23T15:15:22.459+0300] [glassfish 5.0] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1524485720489] [timeMillis: 1524485722459] [levelValue: 800] [[
  Grizzly Framework 2.4.1 started in: 5ms - bound to [/0.0.0.0:4848]]]

[2018-04-23T15:15:22.587+0300] [glassfish 5.0] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1524485720489] [timeMillis: 1524485722587] [levelValue: 800] [[
  Grizzly Framework 2.4.1 started in: 10ms - bound to [/0.0.0.0:3700]]]

[2018-04-23T15:15:22.599+0300] [glassfish 5.0] [INFO] [NCLS-CORE-00017] [javax.enterprise.system.core] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1524485722599] [levelValue: 800] [[
  GlassFish Server Open Source Edition  5.0  (25) startup time : Felix (4 501ms), startup services(2 130ms), total(6 631ms)]]

[2018-04-23T15:15:23.061+0300] [glassfish 5.0] [INFO] [jts.startup_msg] [javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions] [tid: _ThreadID=17 _ThreadName=RunLevelControllerThread-1524485720510] [timeMillis: 1524485723061] [levelValue: 800] [[
  JTS5014: Recoverable JTS instance, serverId = [100]]]

[2018-04-23T15:15:23.671+0300] [glassfish 5.0] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1524485720489] [timeMillis: 1524485723671] [levelValue: 800] [[
  Grizzly Framework 2.4.1 started in: 14ms - bound to [/0.0.0.0:7676]]]

[2018-04-23T15:15:23.724+0300] [glassfish 5.0] [INFO] [NCLS-BOOTSTRAP-00027] [javax.enterprise.bootstrap] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1524485723724] [levelValue: 800] [[
  Registered com.sun.enterprise.glassfish.bootstrap.osgi.EmbeddedOSGiGlassFishImpl@640dc4c6 as OSGi service registration: org.apache.felix.framework.ServiceRegistrationImpl@191a709b.]]

[2018-04-23T15:15:24.709+0300] [glassfish 5.0] [INFO] [NCLS-JMX-00024] [javax.enterprise.system.jmx] [tid: _ThreadID=51 _ThreadName=Thread-13] [timeMillis: 1524485724709] [levelValue: 800] [[
  Creating a SecureRMIServerSocketFactory @ 0.0.0.0 with ssl config = GlassFishConfigBean.org.glassfish.grizzly.config.dom.Ssl]]

[2018-04-23T15:15:24.727+0300] [glassfish 5.0] [INFO] [] [] [tid: _ThreadID=51 _ThreadName=Thread-8] [timeMillis: 1524485724727] [levelValue: 800] [[
  SSLParams =org.glassfish.admin.mbeanserver.ssl.SSLParams@192321c7]]

[2018-04-23T15:15:24.730+0300] [glassfish 5.0] [WARNING] [NCLS-JMX-00020] [javax.enterprise.system.jmx] [tid: _ThreadID=51 _ThreadName=Thread-13] [timeMillis: 1524485724730] [levelValue: 900] [[
  All SSL cipher suites disabled for network-listener(s).  Using SSL implementation specific defaults]]

[2018-04-23T15:15:24.767+0300] [glassfish 5.0] [INFO] [] [] [tid: _ThreadID=51 _ThreadName=Thread-8] [timeMillis: 1524485724767] [levelValue: 800] [[
  SSLParams =org.glassfish.admin.mbeanserver.ssl.SSLParams@192321c7]]

[2018-04-23T15:15:24.771+0300] [glassfish 5.0] [WARNING] [NCLS-JMX-00020] [javax.enterprise.system.jmx] [tid: _ThreadID=51 _ThreadName=Thread-13] [timeMillis: 1524485724771] [levelValue: 900] [[
  All SSL cipher suites disabled for network-listener(s).  Using SSL implementation specific defaults]]

[2018-04-23T15:15:24.851+0300] [glassfish 5.0] [INFO] [NCLS-JMX-00025] [javax.enterprise.system.jmx] [tid: _ThreadID=51 _ThreadName=Thread-13] [timeMillis: 1524485724851] [levelValue: 800] [[
  SSLServerSocket /0.0.0.0:8686 and [SSL: ServerSocket[addr=/0.0.0.0,localport=8686]] created]]

[2018-04-23T15:15:25.058+0300] [glassfish 5.0] [WARNING] [] [org.glassfish.grizzly.filterchain.DefaultFilterChain] [tid: _ThreadID=45 _ThreadName=admin-listener(4)] [timeMillis: 1524485725058] [levelValue: 900] [[
  GRIZZLY0013: Exception during FilterChain execution
java.lang.NoClassDefFoundError: sun/security/ssl/SupportedEllipticCurvesExtension
        at sun.security.ssl.HelloExtensions.<init>(HelloExtensions.java:82)
        at sun.security.ssl.HandshakeMessage$ClientHello.<init>(HandshakeMessage.java:362)
        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:223)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:984)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:924)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:921)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1379)
        at org.glassfish.grizzly.ssl.SSLUtils.executeDelegatedTask(SSLUtils.java:274)
        at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:708)
        at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:622)
        at org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:334)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at org.glassfish.grizzly.portunif.PUFilter.handleRead(PUFilter.java:231)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:539)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573)
        at java.lang.Thread.run(Thread.java:748)
]]

[2018-04-23T15:15:25.168+0300] [glassfish 5.0] [SEVERE] [] [] [tid: _ThreadID=70 _ThreadName=Thread-9] [timeMillis: 1524485725168] [levelValue: 1000] [[
  Exception in thread "RMI TCP Connection(idle)" ]]

[2018-04-23T15:15:25.170+0300] [glassfish 5.0] [SEVERE] [] [] [tid: _ThreadID=70 _ThreadName=Thread-9] [timeMillis: 1524485725170] [levelValue: 1000] [[
  java.lang.NoSuchMethodError: sun.security.ssl.SSLSessionImpl.<init>(Lsun/security/ssl/ProtocolVersion;Lsun/security/ssl/CipherSuite;Ljava/util/Collection;Ljava/security/SecureRandom;Ljava/lang/String;I)V
        at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:785)
        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:227)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:984)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:919)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:938)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
        at java.io.DataInputStream.readInt(DataInputStream.java:387)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:727)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
]]

[2018-04-23T15:15:25.178+0300] [glassfish 5.0] [WARNING] [NCLS-JMX-00007] [javax.enterprise.system.jmx] [tid: _ThreadID=51 _ThreadName=Thread-13] [timeMillis: 1524485725178] [levelValue: 900] [[
  Cannot start JMX connector JmxConnector config: { name = system, Protocol = rmi_jrmp, Address = 0.0.0.0, Port = 8686, AcceptAll = false, AuthRealmName = admin-realm, SecurityEnabled = true} due to exception java.io.IOException: Cannot bind to URL [rmi://ubnt.kronosglobal.com:8686/jmxrmi]: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]]]

[2018-04-23T15:15:25.184+0300] [glassfish 5.0] [SEVERE] [] [] [tid: _ThreadID=51 _ThreadName=Thread-9] [timeMillis: 1524485725184] [levelValue: 1000] [[
  java.io.IOException: Cannot bind to URL [rmi://ubnt.kronosglobal.com:8686/jmxrmi]: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
        at javax.management.remote.rmi.RMIConnectorServer.newIOException(RMIConnectorServer.java:827)
        at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:432)
        at org.glassfish.admin.mbeanserver.RMIConnectorStarter.start(RMIConnectorStarter.java:320)
        at org.glassfish.admin.mbeanserver.JMXStartupService$JMXConnectorsStarterThread.startConnector(JMXStartupService.java:314)
        at org.glassfish.admin.mbeanserver.JMXStartupService$JMXConnectorsStarterThread.run(JMXStartupService.java:351)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
        at com.sun.jndi.rmi.registry.RegistryContext.rebind(RegistryContext.java:177)
        at com.sun.jndi.toolkit.url.GenericURLContext.rebind(GenericURLContext.java:251)
        at javax.naming.InitialContext.rebind(InitialContext.java:433)
        at javax.naming.InitialContext.rebind(InitialContext.java:433)
        at javax.management.remote.rmi.RMIConnectorServer.bind(RMIConnectorServer.java:642)
        at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:427)
        ... 3 more
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
        javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:307)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338)
        at sun.rmi.registry.RegistryImpl_Stub.rebind(RegistryImpl_Stub.java:147)
        at com.sun.jndi.rmi.registry.RegistryContext.rebind(RegistryContext.java:175)
        ... 8 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at java.io.DataOutputStream.flush(DataOutputStream.java:123)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229)
        ... 12 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
        at sun.security.ssl.InputRecord.read(InputRecord.java:505)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
        ... 19 more
]]

[2018-04-23T15:15:29.413+0300] [glassfish 5.0] [WARNING] [] [org.glassfish.grizzly.filterchain.DefaultFilterChain] [tid: _ThreadID=42 _ThreadName=admin-listener(1)] [timeMillis: 1524485729413] [levelValue: 900] [[
  GRIZZLY0013: Exception during FilterChain execution
java.lang.NoClassDefFoundError: sun/security/ssl/SupportedEllipticCurvesExtension
        at sun.security.ssl.HelloExtensions.<init>(HelloExtensions.java:82)
        at sun.security.ssl.HandshakeMessage$ClientHello.<init>(HandshakeMessage.java:362)
        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:223)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:984)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:924)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:921)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1379)
        at org.glassfish.grizzly.ssl.SSLUtils.executeDelegatedTask(SSLUtils.java:274)
        at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:708)
        at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:622)
        at org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:334)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at org.glassfish.grizzly.portunif.PUFilter.handleRead(PUFilter.java:231)
        at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
        at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
        at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
        at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:539)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573)
        at java.lang.Thread.run(Thread.java:748)
]]

Answer 1

As I already answered in the GitHub issue you raised against Payara Server, the solution here for Payara Server 5.181+ is to upgrade your Java version to at least 8u161/8u162. This is due to a change in internal JDK classes which Grizzly uses for HTTP/2.

GlassFish 5.0 does not have the latest version of Grizzly which is compatible with the latest Java 8 updates. If you are using GlassFish 5.0 and see these errors, you have 2 choices:

1. Downgrade Java to 8u152 or earlier
2. Download the latest GlassFish 5.0.1 nightly, where this should be fixed

Ref: https://github.com/javaee/glassfish/issues/22436#issuecomment-374829430