Kafka running on zookeeper subcontext or chroot

Question

State: We are sharing zookeeper with kafka and several different services, which are using zookeeper for coordination. They are nicely operating on zookeeper subcontext. Looks like this:

/
   /service1/...
   /service2/...
   /brokers/...
   /consumers/...

My question is.. Is it possible to setup kafka to use subcontext? So, the other services can't eventually modify other services subcontext. It would be:

/
   /service1/...
   /service2/...
   /kafka/brokers/...
   /kafka/consumers/...

I saw this syntax in other projects:

zk://10.0.0.1,10.0.0.2/kafka

lets say. So, kafka would see only the brokers and consumers paths and there would be no way to mess up with other subcontext.

I'm afraid kafka is just not supported this format at the time. Other question is, is there a workaround? Like wrap up zookeeper somehow? Any ideas? Or kafka is supposed to use zookeeper exclusively. Is it best practice and we should spawn zookeeper for each project, which is overkill thus zookeeper need ensemble consists atleast of 3 nodes.

Thanks for your answer!

Answer 1

You can use the zk chroot syntax with Kafka, as detailed in the Kafka configuration documentation.

Zookeeper also allows you to add a "chroot" path which will make all kafka data for this cluster appear under a particular path. This is a way to setup multiple Kafka clusters or other applications on the same zookeeper cluster. To do this give a connection string in the form hostname1:port1,hostname2:port2,hostname3:port3/chroot/path which would put all this cluster's data under the path /chroot/path. Note that you must create this path yourself prior to starting the broker and consumers must use the same connection string.

The best practice is to maintain a single ZooKeeper cluster (at least that is what I've seen). Otherwise you are creating more operational workload for maintaining a good ZK ensemble.

The Kafka documentation on operationalizing ZK sort of recommends having multiple ZKs though:

Application segregation: Unless you really understand the application patterns of other apps that you want to install on the same box, it can be a good idea to run ZooKeeper in isolation (though this can be a balancing act with the capabilities of the hardware).