Leader Not Available Kafka in Console Producer

Question

It could be related to advertised.host.name setting in your server.properties.

What could happen is that your producer is trying to find out who is the leader for a given partition, figures out its advertised.host.name and advertised.port and tries to connect. If these settings are not configured correctly it then may think that the leader is unavailable.

I tried all the recommendations listed here. What worked for me was to go to server.properties and add:

port = 9092
advertised.host.name = localhost 

Leave listeners and advertised_listeners commented out.

What solved it for me is to set listeners like so:

advertised.listeners = PLAINTEXT://my.public.ip:9092
listeners = PLAINTEXT://0.0.0.0:9092

This makes KAFKA broker listen to all interfaces.

I have been witnessing this same issue in the last 2 weeks while working with Kafka and have been reading this Stackoverflow's post ever since.

After 2 weeks of analysis i have deduced that in my case this happens when trying to produce messages to a topic that doesn't exist.

The outcome in my case is that Kafka sends an error message back but creates, at the same time, the topic that did not exist before. So if I try to produce any message again to that topic after this event, the error will not appear anymore as the topic as been created.

PLEASE NOTE: It could be that my particular Kafka installation was configured to automatically create the topic when the same does not exist; that should explain why in my case I can see the issue only once for every topic after resetting the topics: your configuration might be different and in that case you would keep receiving the same error over and over.

Regards,

Luca Tampellini