Deploy mongodb replicaset servers with Docker on different physical servers

Question

I'm trying to deploy a mongodb replicaset using docker. I managed to do it on a same server by executing this :

docker run -d --expose 27017 --name mongodbmycompany1 dockerfile/mongodb mongod --replSet rsmycompa
docker run -d --expose  27017 --name mongodbmycompany2 dockerfile/mongodb mongod --replSet rsacommeassure
docker run -d --expose 27017 --name mongodbmycompany3 dockerfile/mongodb mongod --replSet rsacommeassure

MONGODB1=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' mongodbmycompany1)
MONGODB2=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' mongodbmycompany2)
MONGODB3=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' mongodbmycompany3)

echo $MONGODB1
echo $MONGODB2
echo $MONGODB3

echo "Mongodb Replicaset init"
docker exec mongodbmycompany1 mongo 127.0.0.1:27017/mycompany --eval 'if(!rs.conf()) {     rs.initiate(); cfg = rs.conf(); cfg.members[0].host = "'$MONGODB1':27017"; rs.reconfig(cfg); rs.add("'$MONGODB2':27017"); rs.add("'$MONGODB3':27017"); } rs.status();'

It's working as expected. My replicaset is initialized and my mongodb resultset config contains my 3 servers identified by their internal IP address. It's not perfect as I'd prefer to use servers names but I didn't manage to do it. Docker only populate each /etc/hosts file with servers names passed at image launch with --link parameter. If i add a new server while others are running. Those servers won't ping the new server.

Now I have another question. In production, having a lot of Mongodb docker image running on a same physical server is possible but it's not safe : - if my physical server falls down, i lose all my Mongodb replicas and my service is down - if my physical server uses internal storage, all my docker images use the same disk... and I'm going to have IO problems.

So my question is : How can I deploy many mongodb replicas on multiple physical servers ? How those mongodb replicas can communicate with each others (primary and secondaries servers can change) while they are on different servers or even on different datacenters ?

Answer 1

Let's assume:

1. you have 3 different docker hosts (servers), with IPs 10.1.1.101, 10.1.1.102, 10.1.1.103
2. want to deploy a single replica set called rsacommeassure
3. Dockerfiles for mongodb expose port 27017
4. all servers are in a trusted zone and can talk to each other

First let's start mongodb containers on each server (10.1.1.101 ~$ is used for command prompt):

10.1.1.101 ~$ docker run -d -p 27017:27017 --name mongodbmycompany1 dockerfile/mongodb mongod --replSet rsacommeassure
10.1.1.102 ~$ docker run -d -p 27017:27017 --name mongodbmycompany2 dockerfile/mongodb mongod --replSet rsacommeassure
10.1.1.103 ~$ docker run -d -p 27017:27017 --name mongodbmycompany3 dockerfile/mongodb mongod --replSet rsacommeassure

-p 27017:27017 exposes port 27017 on the host IP so mongo is accessible on servers' host IP address.

Then you need to initiate the replica set, so just run this against a mongodb container (I'll pick server1 here):

your_laptop ~$ > mongo --host 10.1.1.101 
MongoDB shell version: 2.6.9
connecting to: test
> rs.initiate()
> cfg = rs.conf()
> cfg.members[0].host = "10.1.1.101:27017"
> rs.reconfig(cfg)
> rs.add("10.1.1.102:27017")
> rs.add("10.1.1.103:27017") 
> rs.status();

The IPs are local but it works with global as well as long as the servers can talk to each other (VPN, firewall, DMZ, whatever). Btw you should consider security carefully.

Answer 2

I've created a Replica Set on different physical servers using docker-machine and virtual box as a driver:

$ docker-machine create --driver virtualbox server1
$ docker-machine create --driver virtualbox server2
$ docker-machine create --driver virtualbox server3

Open 3 different terminals, in each

$(Terminal1) eval "$(docker-machine env server1)"
$(Terminal2) eval "$(docker-machine env server2)"
$(Terminal3) eval "$(docker-machine env server3)"

In each terminal:

$(Terminal1) docker run -d -p 27017:27017 --name mongoClient1 mongo mongod --replSet r1
$(Terminal2) docker run -d -p 27017:27017 --name mongoClient2 mongo mongod --replSet r1
$(Terminal3) docker run -d -p 27017:27017 --name mongoClient3 mongo mongod --replSet r1

Go in VirtualBox -> on each environment(server1,server2,server3) -> Setting -> Network -> Adapter 1 -> Port Forwarding. Create a new rule Protocol TCP, Host Port - 27017, Guest Port - 27017, leave Host Ip and Guest Ip empty

Now restart all the environments, you can do this from the VirtualBox or from the terminal, from terminal just run:

$(Terminal1) docker-machine restart server1
$(Terminal2) docker-machine restart server2
$(Terminal3) docker-machine restart server3

Restart the containers:

$(Terminal1) docker start mongoClient1
$(Terminal2) docker start mongoClient2
$(Terminal3) docker start mongoClient3

Now the containers should be running, you can check them by running $ docker ps in each terminal

Get into the first container's(or another) Mongo Shell

$(Terminal1) docker exec -it mongoClient1 mongo
// now we are in the Mongo Shell
$(Mongo Shell) rs.initiate()
$(Mongo Shell) cfg = rs.conf()
$(Mongo Shell) cfg.members[0].host = <server1's Ip Address>
// you should get server1's Ip Address by running $ docker-machine ls, mine was 192.168.99.100
$(Mongo Shell) rs.reconfig(cfg)
$(Mongo Shell Primary) rs.add("<server2's Ip Address>:27017")
// now we added a Secondary
$(Mongo Shell Primary) rs.add("<server3's Ip Address>:27017", true)
// now we added an Arbiter
$(Mongo Shell Primary) use planes
// now we create a new database
$(Mongo Shell Primary) db.tranporters.insert({name:'Boeing'})
// create a new collection
$(Mongo Shell Primary) db.tranporters.find()
// we obtain the inserted plane

To connect to a Secondary, you can either:

$(Terminal2) docker exec -it mongoClient2 mongo planes
// or
$(Mongo Shell Primary) db = connect ("<server2's Ip Address>:27017/planes")

Now we are in the Mongo Shell of a Secondary

$(Mongo Shell Secondary) rs.slaveOk()
// to allow readings from the Shell
$(Mongo Shell Secondary)db.tranporters.find()
// should return inserted plane

Answer 3

You could use "Weave - the Docker network" to resolve your problem easily.

Weave creates an overlay network that joins containers on different hosts, even at different cloud providers. Weave also supplies a DNS service that lets you find containers by name within the Weave network.