I have a very strange problem when trying to connect locally to Kafka 0.10.0.0 using Python client on CentOS.
My connection options are pretty simple and default:
kafka_consumer = kafka.KafkaConsumer(
bootstrap_servers=['localhost:9092'],
client_id="python-test-consumer"
)
When I manually set listeners option in Kafka's server.properties file like:
listeners=PLAINTEXT://localhost:9092
I get the kafka.errors.NoBrokersAvailable despite the fact that I can still easily connect to Kafka broker server with curl or other linux stuff.
No advertised.listeners or other deprecated advertised options help to solve the problem. Thus, the only state of configuration which is working is one without listeners. What is certainly unacceptable, because we need to setup local cluster somehow.
It seems that solution for this silly problem is simple and is wondering around, but we couldn't figure it ourselves.
In order to access Kafka Brokers from outside the cluster, an additional listener and advertised listener must be configured. Additionally, a specific service per kafka pod will be created. There are two ways of configuring external access. Using LoadBalancer services or using NodePort services.
A client that wants to send or receive messages from the Kafka cluster may connect to any broker in the cluster. Every broker in the cluster has metadata about all the other brokers and will help the client connect to them as well, and therefore any broker in the cluster is also called a bootstrap server.
This may sound silly, but the exact same problem happened to me because of this:
I upgraded to Kafka 0.10.0.0 via brew (Mac package manager). Brew then suggests to run like this one-liner:
$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties; kafka-server-start /usr/local/etc/kafka/server.properties
Instead of how I executed before:
$ zkServer start
$ kafka-server-start /usr/local/etc/kafka/server.properties
The approach suggested kept throwing those "No Brokers Available" errors in the client. Then I just split the command in two lines:
$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
$ kafka-server-start /usr/local/etc/kafka/server.properties
And everything works like before!
Sorry if this doesn't work for you, but I figured it was worth mentioning.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With