Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

docker-compose healthcheck does not work in a way it is expected for making container a run first and then container B

I am using docker compose to run couple of service which depends on each other. Here is part of the docker-compose:

  backend:
    build: .
    command: bash -c "npm run build && npm start"
    ports: 
      - "3015:3015"
    depends_on:
      - couchdb
      - redis
      - uds-mock-server
    volumes:
      - /app/node_modules
      - .:/app
    user: root
  api-test:
    restart: always 
    build: .
    depends_on:
      - backend
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3015/readiness"]
      interval: 200s
      timeout: 200s
      retries: 5
    user: root

As you see I have two service over there and backend should first run and the server needs to be ready then api-test can start. backend has an endpoint: localhost:2015/readiness and whenever it returns 200 then api test can start. When I run while building the order is respected so backend first followed by api-mock but when docker compose starts running them api-test run quicker and since it relies on the backend to be ready it fails.

Base on the following:

Docker Compose wait for container X before starting Y

and

Docker healthcheck in composer file

It is suggested that I should use healthcheck which I do in api test:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:3015/readiness"]
  interval: 200s
  timeout: 200s
  retries: 5

If I get this correctly api-test should run and call the readiness endpoint and wait until it hears from backend readiness (up to 200s) if it fails it waits for 200s and then try again for 5 times. But what I see is api-test keep failing and restarting and does not even give a chance to backend to run and it keeps doing this like a loop. Am I missing anything? Any help is really appreciated

like image 516
Learner Avatar asked Nov 27 '19 03:11

Learner


People also ask

What happens when Docker Healthcheck fails?

If a health check fails but the subsequent one passes, the container will not transition to unhealthy . It will become unhealthy after three consecutive failed checks. --timeout – Set the timeout for health check commands. Docker will treat the check as failed if the command doesn't exit within this time frame.

What is healthcheck in Dockerfile?

Docker Dockerfiles HEALTHCHECK Instruction The HEALTHCHECK instruction tells Docker how to test a container to check that it is still working. This can detect cases such as a web server that is stuck in an infinite loop and unable to handle new connections, even though the server process is still running.

What is Docker compose depends?

depends_on is a Docker Compose keyword to set the order in which services must start and stop. For example, suppose we want our web application, which we'll build as a web-app image, to start after our Postgres container.

What is compose file in Docker?

The Compose file provides a way to document and configure all of the application's service dependencies (databases, queues, caches, web service APIs, etc). Using the Compose command line tool you can create and start one or more containers for each dependency with a single command ( docker-compose up ).


Video Answer


2 Answers

The two examples are based on the condition form of depends_on which is no longer supported in compose version 3. So, unless your docker-compose version is <3 the healthcheck will not help you much. The healthcheck sets the status of the container (starting, healthy or unhealthy) but docker-compose does not wait until backend container is healthy before starting the app-test. There is a detailed explanation about how depends_on works in Control startup and shutdown order in Compose

As a side note, the healthcheck in your compose file sets the status of the app-test container and not backend.

Therefore to control when the api-test can start, you have to wrap the service command of the container. For your particular case the following will do the job:

bash -c 'while [[ "$(curl --connect-timeout 2 -s -o /dev/null -w ''%{http_code}'' https://backend:3015/readiness)" != "200" ]]; do echo ..; sleep 5; done; echo backend is up; <service_command>'

It tries to connect to backend every 5 seconds (the connection timeout is 2s). When the received HTTP status code is 200 OK the loop ends and it executes the <service_command>

The relevant docker-compose part:

  api-test:
    restart: always
    command: bash -c 'while [[ "$$(curl --connect-timeout 2 -s -o /dev/null -w ''%{http_code}'' uds-mock-server:4000/readiness)" != "200" ]]; do echo ..; sleep 5; done; echo backend is up;npm start'
    depends_on:
      - backend
      ...

Hope this helps.

like image 81
b0gusb Avatar answered Sep 21 '22 15:09

b0gusb


it's not localhost connection string.

It should be service name of backend container :

test: ["CMD", "curl", "-f", "http://backend:3015/readiness"]
like image 45
Thanh Nguyen Van Avatar answered Sep 17 '22 15:09

Thanh Nguyen Van