Why can't Kafka Producer connect to zookeeper to fetch broker metadata instead of connecting to brokers

Question

Kafka Producers need a bootstrap list of kafka brokers to work. As per this explanation , it needs this to be able to connect to one of the brokers and then fetch metadata about all live brokers in the cluster.

Now, all the brokers are already registered in the Zookeeper and Kafka consumers connect to ZK which handles from which broker , which partition is data to be read from. Why can't Producers also connect to ZK when ZK already has all the information?

I see there are a few SO questions on this but they seem to explain why consumer needs ZK & not why Producer needs bootstrap list of brokers instead of ZK?

Answer 1

Both producers and consumers used to connect to Zookeeper for coordination in the past but has now evolved away from that. With the new consumer API in Kafka 0.9 neither of the clients need to know about Zookeeper anymore and this seems to be the future for Kafka clients. Apart from getting rid of dependencies and flaky connection libraries to ZK, it also makes evolution of the client protocol easier since this is now completely managed by Kafka.

Currently both consumer clients are available in Kafka 0.9 but you should be prepared that the consumer client using ZK connectivity will probably become deprecated at some point in a future release.