docker-compose up --exit-code-from exits without error when system under test crashes fast

Question

I am creating some integration tests for my docker images using docker-compose. docker-compose has a neat flag --exit-code-from to allow stopping all dockers and returning exit code from the test docker once the tests are finished.

Trouble is however that --exit-code-from includes automatically the flag --abort-on-container-exit. This is quite logical, but creates the following problem:

Normal case

1. system under test starts
2. integration-tests run tests and exit with error code X
3. all containers are stopped
4. exit code X is returned.

Problem case

1. system under test starts
2. before integration-tests has the time to finish, system under test exits with an error
3. all containers are stopped
4. exit code 0 is returned! So it is as if the tests succeeded.

You can reproduce this with the following files:

Dockerfile

FROM alpine as development
ENTRYPOINT [ "/bin/false" ]

FROM alpine as test
ENTRYPOINT [ "/bin/sleep", "3" ]

docker-compose.yml

version: "3.5"
services:
  sut:
    build:
      context: .
      target: development
  test:
    build:
      context: .
      target: test

and then running

docker-compose up --build --exit-code-from test
echo $? # print exit code from previous command

which will return exit code 0.

I would like exit code 0 to mean: integration tests really ran and were successful.

EDIT: after seeing this issue https://github.com/docker/compose/pull/6077/commits merged into docker-compose 1.22.0; I ran again with the latest version of docker-compose (1.24.1 as of now), but I run into my problem still.

Answer 1

This is the only workaround I have found so far. Not the cleanest solution, but it does the trick:

docker-compose up --build --exit-code-from test | grep -q "_test_1 exited with code 0" $TMP

a longer version in a clean bash script with option parsing, showing the output of docker-compose: https://gist.github.com/chrismaes87/7297d34d356b07a00a5da5f8e425326c