Kafka Controller could not connect to brokers

Question

I've a 3 node Kafka cluster(version 0.10.1.0). I've followed the steps on kafka security documentation. Here are the related configurations of one of my Kafka servers.

listeners=SSL://myhostname:9093
security.inter.broker.protocol=SSL
advertised.listeners=SSL://myhostname:9093
# In order to enable hostname verification
ssl.endpoint.identification.algorithm=HTTPS

ssl.client.auth=required

# certificate file locations
ssl.keystore.location=/location/server1.keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit
ssl.truststore.location=/location/server.truststore.jks
ssl.truststore.password=changeit

# Supported TLS versions
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

I've defined 3 different keystores for all my Kafka servers and signed them with the same CA. When I start the Kafka servers the controller log constantly logs the following warning log.

WARN [Controller-0-to-broker-2-send-thread], Controller 0's connection to broker host3:9093 (id: 2 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to host3:9093 (id: 2 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93)
    at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:230)
    at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
WARN [Controller-0-to-broker-0-send-thread], Controller 0's connection to broker host1:9093 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to host1:9093 (id: 0 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93)
    at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:230)
    at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
WARN [Controller-0-to-broker-1-send-thread], Controller 0's connection to broker host2:9093 (id: 1 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to host2:9093 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93)
    at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:230)
    at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)

It seems to me more serious than a warning.

Do you have any idea what may be the problem?

Thanks in advance.

Answer 1

I identified the problem, it is about the certificate creation. Refer to Confluent's documentation it says:

Ensure that common name (CN) matches exactly with the fully qualified domain name (FQDN) of the server. The client compares the CN with the DNS domain name to ensure that it is indeed connecting to the desired server, not a malicious one.

I regenerate the certificates and it worked!