ZooKeeper max number of children per node

Question

I am using ZooKeeper to store hierarchical data. In one test I stored more than 300K children for under one node. ZK client crashed due to a ConnectionLossException when I tried to retrieve ALL children.

Remodeling the data might solve the problem (e.g. by bucketing as it reduces the number of children per node).

However I am curious to know the answers of the following questions:

1. ZK has a limitation of data that can stored on one node (no more than 1MB), is there a similar restriction on the number of children per node?
2. Does increasing Java Heap size will allow ZK to scale out the number of children per node?
3. Is there anyway to control the way these children are sent back to client? if the children are sent in one batch then a network "hiccup" could destroy the message and cause failure.

Thanks!

Answer 1

There's no explicit limit to the number of children.

However, there is a limit to the size of the packet that can be sent back from the server in response to getChildren which effectly ends up being the limit. This is controlled with a system property, jute.maxbuffer and defaults to 4MB (I think you can get a around 200,000 znodes into this, though it also depends on the length of the znode name).

Check out this link for the full discussion.

Answer 2

Just a day ago, there was an issue on Zookeeper version 3.4.4 where nearly 800K nodes ended up under a single parent node and was causing issues when trying to access that parent node. Tried a few different methods found for deleting a node, of which none of them worked. All hit ConnectionLoss KeeperExceptions.

The -Djute.maxbuffer java option when run with a java8 program that uses ZKUtil.deleteRecursive( zk, path ); via the 3.4.14 java client (not the 3.4.4 java client) worked to delete the 800K children znodes. This jopt did not have to be placed on the server side. It was a purely client side change. Did not work with the 3.4.14 zkCli command line client however.

Answer 3

Through the zookeeper source , there's no explicit limit to child count . But the childCount is int type , the max should is Integer#MAX_VALUE .