X-pack failed to initialize a TrustManagerFactory

Question

I have set up Elasticsearch, Kibana and X-pack according to installation guidelines and made sure that it worked as expected. Now I want to send a Kibana report using Watchers in X-Pack. I have followed this tutorial to setup secure reporting, but after adding the watcher truststore to elasticsarch.yml I get this error message when I try to start elastic:

[2016-11-22T12:19:57,111][INFO ][o.e.e.NodeEnvironment    ] [CBeNcdh] using [1] data paths, mounts [[OS (c:)]], net usable_space [72.3gb], net total_space [223.5gb], spins? [unknown], types [NTFS]
[2016-11-22T12:19:57,115][INFO ][o.e.e.NodeEnvironment    ] [CBeNcdh] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-11-22T12:19:57,354][INFO ][o.e.n.Node               ] [CBeNcdh] node name [CBeNcdh] derived from node ID; set [node.name] to override
[2016-11-22T12:19:57,358][INFO ][o.e.n.Node               ] [CBeNcdh] version[5.0.1], pid[15948], build[080bb47/2016-11-11T22:08:49.812Z], OS[Windows 10/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_111/25.111-b14]
[2016-11-22T12:19:58,618][ERROR][o.e.b.Bootstrap          ] Exception
org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:462) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:414) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:144) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.node.Node.<init>(Node.java:281) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.node.Node.<init>(Node.java:220) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286) [elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:112) [elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:103) [elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) [elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) [elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.cli.Command.main(Command.java:62) [elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:73) [elasticsearch-5.0.1.jar:5.0.1]
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:451) ~[elasticsearch-5.0.1.jar:5.0.1]
    ... 14 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
    at org.elasticsearch.xpack.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:57) ~[?:?]
    at org.elasticsearch.xpack.ssl.SSLService.createSslContext(SSLService.java:387) ~[?:?]
    at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:423) ~[?:?]
    at org.elasticsearch.xpack.ssl.SSLService.<init>(SSLService.java:78) ~[?:?]
    at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:181) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:451) ~[elasticsearch-5.0.1.jar:5.0.1]
    ... 14 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\tmp\watcher-truststore.jks" "read")
    at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_111]
    at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_111]
    at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_111]
    at java.lang.SecurityManager.checkRead(SecurityManager.java:888) ~[?:1.8.0_111]
    at sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:293) ~[?:?]
    at sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:162) ~[?:?]
    at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:225) ~[?:?]
    at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_111]
    at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_111]
    at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_111]
    at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_111]
    at org.elasticsearch.xpack.ssl.CertUtils.trustManager(CertUtils.java:162) ~[?:?]
    at org.elasticsearch.xpack.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:55) ~[?:?]
    at org.elasticsearch.xpack.ssl.SSLService.createSslContext(SSLService.java:387) ~[?:?]
    at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:423) ~[?:?]
    at org.elasticsearch.xpack.ssl.SSLService.<init>(SSLService.java:78) ~[?:?]
    at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:181) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:451) ~[elasticsearch-5.0.1.jar:5.0.1]
    ... 14 more
[2016-11-22T12:19:58,626][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]]; nested: InvocationTargetException; nested: ElasticsearchException[failed to initialize a TrustManagerFactory]; nested: AccessControlException[access denied ("java.io.FilePermission" "C:\tmp\watcher-truststore.jks" "read")];
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:116) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:103) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:96) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.cli.Command.main(Command.java:62) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:73) ~[elasticsearch-5.0.1.jar:5.0.1]
Caused by: org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:462) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:414) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:144) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.node.Node.<init>(Node.java:281) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.node.Node.<init>(Node.java:220) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:112) ~[elasticsearch-5.0.1.jar:5.0.1]
    ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:451) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:414) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:144) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.node.Node.<init>(Node.java:281) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.node.Node.<init>(Node.java:220) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:112) ~[elasticsearch-5.0.1.jar:5.0.1]
    ... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
    at org.elasticsearch.xpack.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:57) ~[?:?]
    at org.elasticsearch.xpack.ssl.SSLService.createSslContext(SSLService.java:387) ~[?:?]
    at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:423) ~[?:?]
    at org.elasticsearch.xpack.ssl.SSLService.<init>(SSLService.java:78) ~[?:?]
    at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:181) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:451) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:414) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:144) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.node.Node.<init>(Node.java:281) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.node.Node.<init>(Node.java:220) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:112) ~[elasticsearch-5.0.1.jar:5.0.1]
    ... 6 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\tmp\watcher-truststore.jks" "read")
    at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_111]
    at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_111]
    at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_111]
    at java.lang.SecurityManager.checkRead(SecurityManager.java:888) ~[?:1.8.0_111]
    at sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:293) ~[?:?]
    at sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:162) ~[?:?]
    at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:225) ~[?:?]
    at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_111]
    at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_111]
    at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_111]
    at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_111]
    at org.elasticsearch.xpack.ssl.CertUtils.trustManager(CertUtils.java:162) ~[?:?]
    at org.elasticsearch.xpack.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:55) ~[?:?]
    at org.elasticsearch.xpack.ssl.SSLService.createSslContext(SSLService.java:387) ~[?:?]
    at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:423) ~[?:?]
    at org.elasticsearch.xpack.ssl.SSLService.<init>(SSLService.java:78) ~[?:?]
    at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:181) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_111]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:451) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:414) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:144) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.node.Node.<init>(Node.java:281) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.node.Node.<init>(Node.java:220) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286) ~[elasticsearch-5.0.1.jar:5.0.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:112) ~[elasticsearch-5.0.1.jar:5.0.1]
    ... 6 more

I'm running cmd as administrator and the truststore file have the permissions needed for elastic to read it. I do not understand why this error is happening, can anyone please help me?

This is the options added to my elasticsearch.yml file:

xpack.security.http.ssl.truststore.path: C:\tmp\watcher-truststore.jks
xpack.security.http.ssl.truststore.password: <truststorepw>

And my kibana.yml file:

server.ssl.key: C:\tmp\kibana.local.key
server.ssl.cert: C:\tmp\kibana.local.crt

xpack.security.encryptionKey: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"

If I remove the truststore settings from elasticsearch.yml, I can start both elastic and kibana and kibana is running on https.

Answer 1

I posted this question to Elsticsearch forum as well and got the answer to my problem there:

Elasticsearch runs under a security manager with a restricted set of directories that can be read. The key files and certificates should be stored in the config directory and could be placed in a sub directory there if desired