Run multiple docker compose

Question

I am using an application which runs on 3 different docker images:

• The first one is the server HTTP callable with a REST API
• The second one is rabbitmq
• The third one is a worker

The whole application is launched with docker-compose up

Really simple :)

I would like to make it scalable and run multiple instances of this entire application (the 3 docker images) independently of the others, and then put a load balancer like haproxy which will redirect to one of the applications.

I saw that I can use docker-compose up --scale blablabla however the problem with this is that I can scale containers, but I really want to keep the different "applications" independent.

For example if I want 3 versions of the app I will have 9 docker images etc.

I saw that we can run docker inside docker with --privilege (allowing me to create one docker with the 3 dockers inside) but I read on Stack Overflow that it is not a suitable solution.

Do you have a solution? Or at least some documents to read.

I heard that Kubernetes could be a solution but I am not sure. I read (on Stack):

If you need multiple containers tightly bound, you may want to look at Kubernetes that runs docker inside their "pods"

Answer 1

It sounds like you just want 3 different environments or "stacks" of your Compose application running independently of each other. If this is the case, you can handle this with the --project-name or -p option to docker-compose. An example would be something like this:

• Launch application version 1: docker-compose -p appv1 up -d
• Launch application version 2: docker-compose -p appv2 up -d
• Launch application version 3: docker-compose -p appv3 up -d

At this point, you would have 3 different sets of containers running that can scale independently of each other. Docker Compose will prepend the project name (which is usually inferred from the folder name) to the container names. You will end up with container names such as appv1_worker_1, appv2_worker_1, appv3_worker1. If you were to scale only appv2 worker service (docker-compose -p appv2 scale worker=2) you would then get an additional appv2_worker_2.

By default, compose always creates a default network that the containers can talk inside of. In this case, you would have 3 independent networks (appv1_default, appv2_default, and appv3_default).

If you next wanted to run different images for each project name, you could use environment variable interpolation in the docker-compose.yml. For example, you could specify image: ${MYIMAGE} for a service and then do something such as:

• MYIMAGE=myorg/myapp:v1 docker-compose -p appv1 up -d
• MYIMAGE=myorg/myapp:v2 docker-compose -p appv2 up -d

Hope this is helpful as an idea to do it inside of Docker Compose.

Answer 2

If you want to run multiple docker-compose files at once from a single command, use this method. If the compose files are docker-compose1.yml and docker-compose2.yml, you can run both docker files using,

docker-compose -f docker-compose1.yml -f docker-compose2.yml up