Right way to access cadvisor in a swarm environment

Question

I have a Swarm with two nodes and I am running cadvisor as a global service. I get metrics and grafana/Promethues can scrape them. However my numbers are wrong. When I check the number of containers with the docker command line tools I see that there are 17 containers running on the first node and 14 containers on the second node. Prometheus/grafana however will tell me that I got 34 containers and the number of containers is 17 on both nodes.

I assume my problems comes from having cadvisor defined in a different stack than the prometheus and grafana container and trying to access cadvisor via the service API.

In stack A I have

version: '3.6'

services:

  cadvisor:
    image: google/cadvisor
    deploy:
      mode: global
    ports:
      - "8888:8080"
    #privileged: true 
    #command: 
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro    

Then in the prometheus.yml configuration I have the following scrape definition, where host1 and host2 are the host names of the nodes.

  - job_name: 'cadvisor'
    scrape_interval: 20s
    static_configs:
      - targets: ['host1:8888','host2:8888']  

Has someone a working example for using cadvisor in a swarm with more the one node or can point out how to define the targets for cadvisor in the proemtheus configuration?

Answer 1

Maybe a little late, but here it is. To perform the "autodiscovery" way, you may use something like that :

- job_name: 'cadvisor'
  dns_sd_configs:
   - names: ['tasks.cadvisor']
     type: 'A'
     port: 8080 # internal

Within "tasks.cadvisor", the name after the dot must match the name you gave to the service in the docker compose or through the docker service create command.