How can I add a label to the cadvisor and node-exporter metrics?

Question

My node-exporter metrics are something like:

process_cpu_seconds_total{instance="10.1.1.1:8080",job="node_info"}
process_cpu_seconds_total{instance="10.1.1.2:8080",job="node_info"}
process_cpu_seconds_total{instance="10.1.1.15:8080",job="node_info"}

The cadvisor ones:

container_memory_usage_bytes{id="<id>",image="<image>",instance="10.1.1.1:8080",job="docker_info",name="<container name>"}
container_memory_usage_bytes{id="<id>",image="<image>",instance="10.1.1.3:8080",job="docker_info",name="<container name>"}
container_memory_usage_bytes{id="<id>",image="<image>",instance="10.1.1.16:8080",job="docker_info",name="<container name>"}

I want to add a label such as machine_name, something like this:

process_cpu_seconds_total{machine_name="cool_machine",instance="10.1.1.1:8080",job="node_info"}
container_memory_usage_bytes{machine_name="cool_machine",id="<id>",image="<image>",instance="10.1.1.1:8080",job="docker_info",name="<container name>"}

When I try to filter by machine I need to deal with the IP (10.1.1.1), and that is not very user friendly. I wanted to configure node-exporter and cadvisor to add a label to all metrics, this way I can identify the machine no matter what is the IP they have now.

By the way, changing the DNS so the machine answers in another address is not much of an option for me.

My prometheus config is something like:

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'machines_monitor'
scrape_configs:
  - job_name: 'node_info'
    static_configs:
      - targets:
          - 10.1.1.1:8080
          - 10.1.1.2:8080
          - 10.1.1.15:8080
  - job_name: 'docker_info'
    static_configs:
      - targets:
          - 10.1.1.1:8080
          - 10.1.1.3:8080
          - 10.1.1.16:8080

I can create a scrape_configs for machine and start filtering by that but I don't know if that is a good idea, maybe a performance issue with Prometheus.

I'm trying to add labels to the metrics, but I'm other approaches to help to identify the machines are very welcome.

Answer 1

You could try the following:

scrape_configs:
  - job_name: 'node_info'
    static_configs:
      - targets:
          - 10.1.1.1:8080
          - 10.1.1.2:8080
          - 10.1.1.15:8080
    relabel_configs:
      - source_labels: [__address__]
        regex: '10\.1\.1\.1.+'
        replacement: cool_machine_1
        target_label: machine_name
      - source_labels: [__address__]
        regex: '10\.1\.1\.2.+'
        replacement: cool_machine_2
        target_label: machine_name
      ...