Cannot Scrape exposed Docker Metrics from Prometheus container when in custom network (on Linux Host)

Question

We have a Prometheus container and various other services and metrics exporter containers running in a custom docker network.

In our local dev environment on Win 10 x64 using Docker desktop everything works fine, the Prometheus container can scrape the metrics exporter containers in the custom network as well as the exposed Docker metrics from the host (Win 10).

On our Ubuntu 18.04 Linux host, latest Prometheus 2.30.3 container scrapes the metrics exporter containers in the custom network just fine but is unable to scrape the exposed Docker metrics from the host.

These are the specs and config for the Ubuntu host:

OS: Ubuntu 18.04.1 LTS
Prometheus: 2.30.3 (latest)
Docker: 20.10.10 (latest, allows to add extra hosts to container, see below)
Docker-compose: 2.0.1

docker metrics config in /etc/docker/daemon.json

{
  "experimental": true,
  "metrics-addr": "0.0.0.0:9933"
}

config in /prometheus.yml:

...
 - job_name: 'docker_metrics'
    metrics_path: /metrics
    static_configs:
      - targets: ['host.docker.internal:9933']
...

docker-compose config for Prometheus container:

...

prometheus:
    container_name: prom_test
    image: prom/prometheus:v2.30.3
    restart: unless-stopped
    command: "--config.file=/etc/prometheus/prometheus.yml
                  --storage.tsdb.path=/prometheus
                  --storage.tsdb.retention.time=45d"
    ports:
      - 9090:9090
    volumes:
      - ./data/prometheus:/prometheus
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml:ro
    extra_hosts:
      host.docker.internal: host-gateway
    networks:
      - test-network

...

We specify extra hosts in the container config via host.docker.internal: host-gateway, which adds the Linux host's IP address to the container's /etc/hosts file.

When I access Prometheus at https://<my_ip>:9090/targets to inspect the status of the scrape targets, Prometheus ultimately then states:

But, clicking on the http://host.docker.internal:9933/metrics link provided in the scrape target endpoint block actually lists the Docker metrics

jumping into the running Prometheus container and executing cat /etc/hosts inside the container shows that the extra host was added correctly

127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.1      host.docker.internal
172.22.0.2      fa87b35d3923

running wget inside the Prometheus container on the Docker metrics endpoint results in a timeout:

/prometheus $ wget host.docker.internal:9933/metrics
Connecting to host.docker.internal:9933 (172.17.0.1:9933)
wget: can't connect to remote host (172.17.0.1): Connection timed out

The problem - on Linux - seems to be that Prometheus cannot access the host when inside a custom Docker network. When I remove the custom network and set network_mode: host on the Prometheus service in docker-compose, then Prometheus is able to scrape the host Docker metrics but it cannot access the other metrics exporter containers inside the custom network anymore.

How can I collect metrics from the metrics exporter containers (in the custom network) as well as collect host Docker metrics on Linux ?

Thanks !

Answer 1

figured this out:

in our case UFW on Ubuntu was blocking the requests on docker0 interface from our containers that are in a custom docker network.

You can see this by checking the UFW logs, you will see how it blocks the requests for 172.17.0.1 (--> host.docker.internal) on the specified metrics port.

Disabling UFW temporarily made it work, Prometheus could now scrape the Docker metrics on the host.

Re-enabled UFW, then...

(1) gave a custom interface name to my custom docker network

networks:
  my_nw:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.name: my_nw_if

(2) added a UFW rule to allow traffic on the the custom network interface for the metrics port solved the issue sudo ufw allow in on <custom_interface_name> to any port <port>

This now allows the Prometheus container to scrape the metrics on the host on that port.

Thanks to everybody here who chimed in !