Live reload Prometheus configuration in docker(-compose)

Question

I have a new server running Prometheus in docker-compose. I want to be able to re-load the configuration file (prometheus.yml) without have to stop and start the container.

Of course since I persist the storage of promethues in a volume the stop and start isn't really a problem but it seems like overkill, especially since prometheus itself has such a handy api to reload configs.

I see other people with similar questions (e.g. here) but I have been unable to get those solutions to work for me. Maybe I'm overlooking something there.

docker-compose.yml

version: "3"

services:

  grafana:
    restart: always
    container_name: grafana
    image: grafana/grafana:6.2.1
    ports:
      - 3000:3000
    volumes:
      - grafanadata:/var/lib/grafana

  prometheus:
    restart: always
    container_name: prometheus
    image: prom/prometheus:v2.10.0
    privileged: true
    volumes:
      - ./configuration/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheusdata:/prometheus

    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.enable-admin-api'
      - '--web.enable-lifecycle'
    ports:
      - 9090:9090

  node:
    restart: always
    container_name: node
    image: prom/node-exporter:v0.18.0
    ports:
      - 9100:9100

volumes:
  grafanadata:
  prometheusdata:

Alas, my results..

When I run curl -X POST http://localhost:9090/-/reload the docker-compose logs give:

prometheus    | level=info ts=2019-06-17T15:33:02.690Z caller=main.go:730 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
prometheus    | level=info ts=2019-06-17T15:33:02.691Z caller=main.go:758 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml

So the prometheus' end is working fine.. All good so far.

However, when I edit ./configuration/prometheus/prometheus.yml the changes don't propogate to the container. Furthermore, when I try to edit /etc/promethus/prometheus.yml in container I see that it is read only (and as an aside, the container does not have a 'sudo' command).

Is there a docker native way to hot/live reload these config files to the container directory?

As stated, the down/start option works for now but I'm curious if there is a more elegant solution.

Answer 1

docker-compose kill -s SIGHUP prometheus does the trick, so Vishrant was certainly on to something there.