How to setup zookeeper cluster on docker swarm

Question

Environment : 6 server docker swarm cluster (2 master & 4 workers)

Requirement : We need to setup a zookeeper cluster on existing docker swarm.

Blocked on : To setup zookeeper in cluster, we need to provide all zk servers in each server config and provide unique ID in myid file.

Question : When we create a replica of zookeeper in docker swarm, how can we provide unique ID for each replica. Also how can we update zoo.cfg config file with ID of each zookeeper container.

Answer 1

This is not currently an easy ask. Fully scalable stateful application clusters is tricky when each cluster member has a need for a unique identity and storage volume.

On Docker Swarm, today, you are perhaps best advised to run each cluster member as a separate service, in your compose file (See 31z4/zookeeper-docker):

version: '2'
services:
    zoo1:
        image: 31z4/zookeeper
        restart: always
        ports:
            - 2181:2181
        environment:
            ZOO_MY_ID: 1
            ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888

    zoo2:
        image: 31z4/zookeeper
        restart: always
        ports:
            - 2182:2181
        environment:
            ZOO_MY_ID: 2
            ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
..
..

For a state of the art (but still evolving) solution, I recommend checking out Kubernetes:

• https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/

The new concept of Statefulsets offers much promise. I expect Docker Swarm will grow a similar capability in time, where each container instance is assigned a unique and "sticky" hostname, which can be used as the basis for a unique identifier.

Answer 2

We have created a docker image extending the official one which does exact that. The entrypoint.sh has been modified so that on startup of each container, it auto-discovers rest zookeeper nodes and configures the current node appropriately.

You can find the image in the docker store and in our github.

Note: Currently it does not handle cases such as a re-creation of a container cause of a failure.

EDIT (6/11/2018)

The latest image supports re-configuration of zookeeper cluster in cases like:

• Scaling up the docker service (adding more containers)
• Scaling down the docker service (removing containers)
• A container is re-scheduled by docker swarm cause of a failure (new IP is assigned)