Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

docker-compose healthcheck retry frequency != interval

I recently set up healthchecks in my docker-compose config.

It is doing great and I like it. Here's a typical example:

services:
  app:
    healthcheck:
      test: curl -sS http://127.0.0.1:4000 || exit 1
      interval: 5s
      timeout: 3s
      retries: 3
      start_period: 30s

My container is quite slow to boot, hence I set up a 30 seconds start_period.

But it doesn't really fit my expectation: I don't need check every 5 seconds, but I need to know when the container is ready for the first time as soon as possible for my orchestration, and since my start_period is approximative, if it is not ready yet at first check, I have to wait for interval before retry.

What I'd like to have is:

  • While container is not healthy, retry every 5 seconds
  • Once it is healthy, check every 1 minute

Ain't there a way to achieve this out-of-the-box with docker-compose?

I could write a custom script to achieve this, but I'd rather have a native solution if it is possible.

like image 936
Augustin Riedinger Avatar asked Feb 03 '20 11:02

Augustin Riedinger


People also ask

What happens when docker Healthcheck fails?

If a health check fails but the subsequent one passes, the container will not transition to unhealthy . It will become unhealthy after three consecutive failed checks. --timeout – Set the timeout for health check commands. Docker will treat the check as failed if the command doesn't exit within this time frame.

How can I check the health status of a docker container?

We can also see the health status by running docker ps. Notice under STATUS, the status is Up with (healthy) next to it. The health status appears only when a health check is configured.

What does docker Healthcheck do?

These exit codes are how Docker's HEALTHCHECK determines the health of the container. Based on that, we can check the status of the application in the container. Initially, it will take some time to start the health check and update, whether the application is healthy or not.

What is Userns_mode?

There is the userns_mode property which can be used to disable user namespace using userns_mode: "host" The username space feature can be enabled on the docker deamon which will be used by docker-compose when starting the containers.


1 Answers

Unfortunately, this is not possible out of the box.
All the duration set are final. They can't be changed depending on the container state.

However, according to the documentation, the probe does not seem to wait for the start_period to finish before checking your test. The only thing it does is that any failure hapenning during start_period will not be considered as an error.

Below is the sentence that make me think that :

start_period provides initialization time for containers that need time to bootstrap. Probe failure during that period will not be counted towards the maximum number of retries. However, if a health check succeeds during the start period, the container is considered started and all consecutive failures will be counted towards the maximum number of retries.

I encourage you to test if this is really the case as I've never really paid any attention if the healthcheck is tested during the start period or not.
And if it is the case, you can probably increase your start_period if you're unsure about the duration and also increase the interval in order to find a good compromise.

like image 191
Marc ABOUCHACRA Avatar answered Oct 20 '22 21:10

Marc ABOUCHACRA