Kafka server SSL configuration exception

Question

this is one part of my Kafka server.properties configurations:

listeners=SSL://192.168.78.131:9092
ssl.keystore.location=/home/linuxea/encr/server.keystore.jks
ssl.keystore.password=linuxea
ssl.key.password=linuxea
security.inter.broker.protocol=SSL

throw an SSL exception: when I start up the server

[2018-04-18 02:05:32,229] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)\
[2018-04-18 02:05:32,245] ERROR [KafkaServer id=0] Connection to node 0 failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2018-04-18 02:05:32,246] WARN SSL handshake failed (kafka.utils.CoreUtils$)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
    at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
    at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
    at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
    at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:434)
    at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:299)
    at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:253)
    at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:79)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:486)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:424)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460)
    at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:73)
    at kafka.server.KafkaServer.doControlledShutdown$1(KafkaServer.scala:485)
    at kafka.server.KafkaServer.kafka$server$KafkaServer$$controlledShutdown(KafkaServer.scala:534)
    at kafka.server.KafkaServer$$anonfun$shutdown$1.apply$mcV$sp(KafkaServer.scala:556)
    at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:85)
    at kafka.server.KafkaServer.shutdown(KafkaServer.scala:556)
    at kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:48)
    at kafka.Kafka$$anon$1.run(Kafka.scala:89)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
    at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:388)
    at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:468)
    at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:326)
... 13 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
    at sun.security.validator.Validator.validate(Validator.java:260)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
     at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601)
... 22 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
... 28 more

I even not to try to start the client.Actually, I don't know too much about security of kafka, I just configure according to the kafka document. What should I do next step ?

thanks sincerely!

Answer 1

With the properties you have set, the server is going to start in one way SSL mode, to enable two way SSL you have to add ssl.client.auth=required

In latest version of kafka server the Hostname validation is enabled in order to prevent man in the middle attacks, and set to HTTPS by default. You can disable hostname validation by setting:

ssl.endpoint.identification.algorithm=

If you have only one broker then try removing security.inter.broker.protocol=SSL just setting the above string empty will probably solve the problem

Answer 2

You use SSL for inter-broker communication. When the brokers connect and talk to each other they act as clients.

When the brokers connect and do the handshake, the client (= the broker which is opening connection) needs to verify the identity of the server (= the broker which is accepting the connection). The exception you have is basically saying that this failed in your case.

This has to be done using a truststore. You need to create a truststore which should contain either the public key of the CA you used to sign the broker certificates or the public keys of all broker certificates in case you used self-signed certificates. And then specify ssl.truststore.location and ssl.truststore.password options in your broker config file. That should help.