prometheus dynamic metrics_path

Question

Prometheus allows me to dynamically load targets with file_sd_config from a .json file like this

#prometheus.yaml
- job_name: 'kube-metrics'
  file_sd_configs:
  - files:
    - 'targets.json'

[
  {
    "labels": {
      "job": "kube-metrics"
    },
    "targets": [
      "http://node1:8080",
      "http://node2:8080"
    ]
  }
]

However my targets differ in the metrics_path and not the host (I want to scrape metrics for every kubernetes node on <kube-api-server>/api/v1/nodes/<node-name>/proxy/metrics/cadvisor) but I can only set the metrics_path at the job level and not per target. Is this even achievable with prometheus or do I have to write my own code to scrape all these metrics and export them at a single target. Also I couldn't find a list of all supported auto discovery mechanisms, did I miss something in the docs?

Answer 1

You can use relabel_config in Prometheus config to change __metrics_path__ label config.

The principe is to provide the metrics path in your targets under the form host:port/path/of/metrics (note: drop the http://, it is in scheme parameter of scrape_config)

[
  {
    "targets": [
      "node1:8080/first-metrics",
      "node2:8080/second-metrics"
    ]
  }
]

And then replace the related meta-labels with the parts

- job_name: 'kube-metrics'
  file_sd_configs:
  - files:
    - 'targets.json'
  relabel_configs:
    - source_labels: [__address__]
      regex:  '[^/]+(/.*)'            # capture '/...' part
      target_label: __metrics_path__  # change metrics path
    - source_labels: [__address__]
      regex:  '([^/]+)/.*'            # capture host:port
      target_label: __address__       # change target

You can reuse this method on any label known at configuration time to modify the config of the scrape.

On Prometheus, use the service discovery page to check your config has been correctly modified.

The official list of service discovery is in the configuration documentation: look for the *_sd_config in the index.

Answer 2

There is a slightly more elegant way of doing this, that doesn't involve manipulating the address. You can have a label with your target that can be the source of your "relabel" action, something like this:

- labels:
    __meta_discovery_path: '/first-metrics'
  targets:
  - 'node1:8080'
- labels:
    __meta_discovery_path: '/second-metrics'
  targets:
  - 'node2:8080'

And then your relabel would simply be:

  relabel_configs:
    - source_labels: [__meta_discovery_path]
      target_label: __metrics_path__  # change metrics path

Since the label being used starts with "__", it will be stripped from the metrics being pulled, making it nice and clean.