passing multiple .yml files to docker-compose

Question

Docker noob here. I have two files docker-compose.build.yml and docker-compose.up.yml in my docker folder. Following are the contents of both files..

docker-compose.build.yml

version: "3"
services:

    base:
        build:
          context: ../
          dockerfile: ./docker/Dockerfile.base
          args:
            DEBUG: "true"
        image: ottertune-base
        labels:
          NAME: "ottertune-base"

    web:
        build:
          context: ../
          dockerfile: ./docker/Dockerfile.web
        image: ottertune-web
        depends_on:
          - base
        labels:
          NAME: "ottertune-web"
        volumes:
          - ../server:/app

    driver:
        build:
          context: ../
          dockerfile: ./docker/Dockerfile.driver
        image: ottertune-driver
        depends_on:
          - base
        labels:
          NAME: "ottertune-driver"

docker-compose.up.yml

version: "3"
services:

    web:
        image: ottertune-web
        container_name: web
        expose:
          - "8000"
        ports:
          - "8000:8000"
        links:
          - backend
          - rabbitmq
        depends_on:
          - backend
          - rabbitmq
        environment:
          DEBUG: 'true'
          ADMIN_PASSWORD: 'changeme'
          BACKEND: 'postgresql'
          DB_NAME: 'ottertune'
          DB_USER: 'postgres'
          DB_PASSWORD: 'ottertune'
          DB_HOST: 'backend'
          DB_PORT: '5432'
          DB_OPTS: '{}'
          MAX_DB_CONN_ATTEMPTS: 30
          RABBITMQ_HOST: 'rabbitmq'
        working_dir: /app/website
        entrypoint: ./start.sh
        labels:
          NAME: "ottertune-web"
        networks:
          - ottertune-net


    driver:
        image: ottertune-driver
        container_name: driver
        depends_on:
          - web
        environment:
          DEBUG: 'true'
        working_dir: /app/driver
        labels:
          NAME: "ottertune-driver"
        networks:
          - ottertune-net

    rabbitmq:
        image: "rabbitmq:3-management"
        container_name: rabbitmq
        restart: always
        hostname: "rabbitmq"
        environment:
           RABBITMQ_DEFAULT_USER: "guest"
           RABBITMQ_DEFAULT_PASS: "guest"
           RABBITMQ_DEFAULT_VHOST: "/"
        expose:
           - "15672"
           - "5672"
        ports:
           - "15673:15672"
           - "5673:5672"
        labels:
           NAME: "rabbitmq"
        networks:
          - ottertune-net

    backend:
        container_name: backend
        restart: always
        image: postgres:9.6
        environment:
          POSTGRES_USER: 'postgres'
          POSTGRES_PASSWORD: 'ottertune'
          POSTGRES_DB: 'ottertune'
        expose:
          - "5432"
        ports:
          - "5432:5432"
        labels:
          NAME: "ottertune-backend"
        networks:
          - ottertune-net

networks:
   ottertune-net:
      driver: bridge

Nothing wrong with the dockerfiles, i just have a few doubts about this approach.

1. What purpose does having multiple files serve instead of just one docker-compose.yml?
2. How does docker-compose work when used with multiple files?
3. When i do docker-compose -f docker-compose.build.yml build --no-cache

Building base
Step 1/1 : FROM ubuntu:18.04
---> 775349758637
[Warning] One or more build-args [DEBUG] were not consumed
Successfully built 775349758637
Successfully tagged ottertune-base:latest
Building web
Step 1/1 : FROM ottertune-base
---> 775349758637
Successfully built 775349758637
Successfully tagged ottertune-web:latest 
Building driver
Step 1/1 : FROM ottertune-base
---> 775349758637
Successfully built 775349758637
Successfully tagged ottertune-driver:latest 

and then docker-compose up i get the error

rabbitmq is up-to-date                                                                                                                                                                       
backend is up-to-date                                                                                                                                                                        Starting web ... error                                                                                                                                                                                                                                                                                                                                                                    
ERROR: for web  Cannot start service web: OCI runtime create failed: container_linux.go:346: 
starting container process caused "exec: \"./start.sh\": stat ./start.sh: no such file or 
directory": unknown                                                                                                                                                                                                                                                                                                                                                                             
ERROR: for web  Cannot start service web: OCI runtime create failed: container_linux.go:346: 
starting container process caused "exec: \"./start.sh\": stat ./start.sh: no such file or 
directory": unknown                                                                                                                                                                                
ERROR: Encountered errors while bringing up the project.

this entrypoint start.sh is defined in the docker-compose.up.yml file which I didn't pass as an argument to

docker-compose build

So, why is the docker-compose up trying to run this entrypoint from a yml file which is not even passed during build? Really confused on this and didn't find much about it on google and stackoverflow.

Answer 1

If you docker-compose -f a.yml -f b.yml ..., Docker Compose merges the two YAML files. If you look at the two files you've posted, one has all of the run-time settings (ports:, environment:, ...), and if you happened to have the images already it would be enough to run the application. The second only has build-time settings (build:), but requires the source tree checked out locally to be able to run.

You probably need to specify both files on every docker-compose invocation

docker-compose -f docker-compose.build.yml -f docker-compose.up.yml up --build

It does seem like the author of these files intended for them to be run separately

docker-compose -f docker-compose.build.yml build
docker-compose -f docker-compose.up.yml up

but note that some of the run-time options in the build file, like volumes: to hide the application built into the image, will never take effect.

(You should be able to delete a large number of settings in the "up" YAML file that either duplicate what's in the image or that Docker Compose can provide for you: container_name:, expose:, links:, working_dir:, entrypoint:, networks:, and (probably) labels: are all unnecessary and can be deleted.)

Answer 2

What purpose does having multiple files serve instead of just one docker-compose.yml?

You can share configuration across environments. For example, I keep the common configuration such as the network and server in a docker-compose.yml. I keep my development environment specifics such as a server with automatic reload and debugging enabled in a docker-compose.override.yml. I keep the production-specific configs in a docker-compose.prod.yml. Then I can run docker-compose up --build for my development environment (Docker Compose uses docker-compose.yml and docker-compose.override.yml by default). And I can run my prod environment with docker-compose -f docker-compose.yml -f docker-compose.prod.yml up --build. You can read about this in the dedicated docs page.

How does docker-compose work when used with multiple files?

It takes the first file as the base file, and adds or replaces configs from subsequent files ot the base file. See the relevant docs.

When i do docker-compose -f docker-compose.build.yml build --no-cache ...

As for your last question, I can't really tell by what I've seen. But unlike Dockerfiles which need two commands (docker build and docker run), docker-compose only needs one. So when you do docker-compose up, it looks for a file named docker-compose.yml (and also docker-compose.override.yml if it's present).