Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

No Brokers Available error when trying to connect to Kafka

I have a very strange problem when trying to connect locally to Kafka 0.10.0.0 using Python client on CentOS.

My connection options are pretty simple and default:

kafka_consumer = kafka.KafkaConsumer(
        bootstrap_servers=['localhost:9092'],
        client_id="python-test-consumer"
    )

When I manually set listeners option in Kafka's server.properties file like:

listeners=PLAINTEXT://localhost:9092

I get the kafka.errors.NoBrokersAvailable despite the fact that I can still easily connect to Kafka broker server with curl or other linux stuff.

No advertised.listeners or other deprecated advertised options help to solve the problem. Thus, the only state of configuration which is working is one without listeners. What is certainly unacceptable, because we need to setup local cluster somehow.

It seems that solution for this silly problem is simple and is wondering around, but we couldn't figure it ourselves.

like image 563
Andrey Lifanov Avatar asked Aug 02 '16 15:08

Andrey Lifanov


People also ask

How do I access Kafka broker?

In order to access Kafka Brokers from outside the cluster, an additional listener and advertised listener must be configured. Additionally, a specific service per kafka pod will be created. There are two ways of configuring external access. Using LoadBalancer services or using NodePort services.

Does Kafka client connect to all brokers?

A client that wants to send or receive messages from the Kafka cluster may connect to any broker in the cluster. Every broker in the cluster has metadata about all the other brokers and will help the client connect to them as well, and therefore any broker in the cluster is also called a bootstrap server.


1 Answers

This may sound silly, but the exact same problem happened to me because of this:

I upgraded to Kafka 0.10.0.0 via brew (Mac package manager). Brew then suggests to run like this one-liner:

$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties; kafka-server-start /usr/local/etc/kafka/server.properties

Instead of how I executed before:

$ zkServer start
$ kafka-server-start /usr/local/etc/kafka/server.properties

The approach suggested kept throwing those "No Brokers Available" errors in the client. Then I just split the command in two lines:

$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
$ kafka-server-start /usr/local/etc/kafka/server.properties

And everything works like before!

Sorry if this doesn't work for you, but I figured it was worth mentioning.

like image 156
Maximiliano Guerra Avatar answered Sep 30 '22 11:09

Maximiliano Guerra