Prometheus node_exporter in docker: Host networking vs hostnames

Question

To monitor a small home server, I run prometheus and node_exporter (and grafana, and several more things) on docker (similar to https://github.com/stefanprodan/dockprom). I run prometheus on a bridged docker network. For node_exporter, I have two options, which affect the node_network_transmit_bytes_total metric.

• Using the same bridged docker network as prometheus
  • Pro: nodeexporter can be addressed directly by name, thanks to docker's internal DNS
  • Con: The node_network_transmit_bytes_total metric only has docker's virtual internal NIC, not the physical NIC of the box being monitored. This is depsite bind-mounting /proc from the host into /host/proc in the container (specifically my physical interface is eno0, visible in /proc/net/dev on the host):

    $ docker exec -it nodeexporter2 cat /host/proc/net/dev | awk '{print $1}'
    Inter-|
    face
    eth0:
    lo:
    

• Using host-mode networking for nodeexporter
  • Pro: All NICs, including the physical host NIC, are visible
  • Con: There does not appear to be a clean way for prometheus to address nodeexporter:
    • localhost means prometheus itself
    • Hostname of host seems inaccessible? Running docker exec -it prometheus wget -O - http://actual-hostname:9100/metrics works (and uses my host's LAN IP, 192.168.x.x), but configuring actual-hostname:9100 as a prometheus target gives an error (Get "http://actual-hostname:9100/metrics": dial tcp 127.0.1.1:9100: connect: connection refused). I'm not sure why they're resolving differently.
    • What I ended up doing is emulating the host.docker.internal feature available for docker-on-windows and docker-on-mac, by adding this to my docker-compose.yml:

      extra_hosts:
      - "host.docker.internal:172.18.0.1"
      

      That's very brittle, however: That 172.18 was just recently 172.19; I believe it changed on reboot or docker version upgrade. I'd love to do be able to set the extra_hosts to a result of running some script on the host to determine the correct network name, but that wouldn't automatically be re-run on boot.

Any advice?

Answer 1

Late edit: Thanks to thomas, turns out there's a magic host host-gateway that does this, so extra_hosts: ["host.docker.internal:host-gateway"] should do the trick. undocumented, but apparently it's implemented here. and already live in docker 20.10.6 (and likely earlier).

---

I ended up solving this by manually configuring the network:

networks:
  monitor-net:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.23.0.0/16
          ip_range: 172.23.5.0/24
          gateway: 172.23.5.254

# ...

services:
  nodeexporter:
    network_mode: host
    # ...
  prometheus:
    networks:
      - monitor-net
    extra_hosts:
      - "host.docker.internal: 172.23.5.254"

Then prometheus has the target of host.docker.internal for node_exporter, and the address should be stable.