Relabel instance to hostname in Prometheus

Question

I have Prometheus scraping metrics from node exporters on several machines with a config like this:

scrape_configs:   - job_name: node_exporter     static_configs:       - targets:         - 1.2.3.4:9100         - 2.3.4.5:9100         - 3.4.5.6:9100 

When viewed in Grafana, these instances are assigned rather meaningless IP addresses; instead, I would prefer to see their hostnames. I think you should be able to relabel the instance label to match the hostname of a node, so I tried using relabelling rules like this, to no effect whatsoever:

relabel_configs:   - source_labels: ['nodename']     target_label: 'instance' 

I can manually relabel every target, but that requires hardcoding every hostname into Prometheus, which is not really nice. I see that the node exporter provides the metric node_uname_info that contains the hostname, but how do I extract it from there?

node_uname_info{domainname="(none)",machine="x86_64",nodename="myhostname",release="4.13.0-32-generic",sysname="Linux",version="..."} 1 

Answer 1

I juste came across this problem and the solution is to use a group_left to resolve this problem. You can't relabelled with a non existant value in the request, you are limited to the different parameters that you gave to prometheus or those that exists in the module use for the request (gcp,aws...).

So the solution I used is to combine an existing value containing what we want (the hostnmame) with a metric from the node exporter. Our answer exist inside the node_uname_info metrics wich contain the nodename value.

I used the answer to this post as a model for my request: https://stackoverflow.com/a/50357418 .

The solution is this one:

node_memory_Active * on(instance) group_left(nodename) (node_uname_info) 

With this, the node_memory_Active metrics wich contains only instance and job by default as a third value nodename that you can use in the description field of grafana.

Hope that this will help others.

Answer 2

I found hardcode solution:

      global:       scrape_interval: 5s       scrape_timeout: 5s       external_labels:         monitor: 'Prometheus'      scrape_configs:      - job_name: 'shelby'       static_configs:       - targets:         - 10.100.0.01:9100       relabel_configs:       - source_labels: [__address__]         regex: '.*'         target_label: instance         replacement: 'shelby'      - job_name: 'camaro'       static_configs:       - targets:         - 10.101.0.02:9100       relabel_configs:       - source_labels: [__address__]         regex: '.*'         target_label: instance         replacement: 'camaro'      - job_name: 'verona'       static_configs:       - targets:         - 10.101.0.03:9100       relabel_configs:       - source_labels: [__address__]         regex: '.*'         target_label: instance         replacement: 'verona'  

Result:

      node_load15{instance="camaro",job="camaro"}    0.16     node_load15{instance="shelby",job="shelby"}    0.4     node_load15{instance="verona",job="verona"}    0.07