Can the same Zookeeper instance be used by number of services?

Question

Is one Zookeeper installation good enough to be used by Hadoop Kafka and Storm clusters? I want to deploy all on one test environment and try playing with those technologies, can I use one zookeeper installation for that? same znode could be dedicated for number of services?

Answer 1

Yes, you can use a single zookeeper installation to support more than one cluster and indeed different types of clusters. This has been the case for a long time - here's a link to a good discussion on it from 2009: http://zookeeper-user.578899.n2.nabble.com/Multiple-ZK-clusters-or-a-single-shared-cluster-td3277547.html

For testing this is fine (and even to run it on one ZK server). For production use though you'll want at least a 3 node cluster. And you should think carefully about running everything off of a single cluster.

The reason is that if you run multiple Hadoop, Storm and Kafka clusters off of a single ZK cluster, that one set of servers becomes a single point of failure for all of your distributed systems. You can beef up the ZK setup with more than 3 servers (let's say 7) so that it can handle multiple failures, but if someone were to accidentally bring ZK down all your distributed environments would come down too.

Some argue that you would be better off with more isolation between systems. I think it varies by use case but I'd be careful about putting all of your eggs in one ZK basket.