Prometheus: How to replace Consul server ports with regex?

Question

When Prometheus uses Consul's auto-discovery feature to get the list of targets to monitor, it also gets Consul servers themselves. This is great - we want to monitor these guys with Prometheus. The problem is that Consul reports these nodes with port 8300, which is not the port we use to monitor targets.

How does one replace ports received from Consul to something else? I know this is possible with Prometheus relabel_configs but I've yet to successfully configure it.

Answer 1

I've eventually figured it out. below is a working example. As the documentation specifies, the address keyword might not work for all setups - you might want to try "<__meta_consul_address>:<__meta_consul_service_port>" instead.

   - source_labels: ['__address__']
     separator:     ':'
     regex:         '(.*):(8300)'
     target_label:  '__address__'
     replacement:   '${1}:9126'

Answer 2

In case you would like to dynamically define port by additional annotation, ie.:

  selector:
    matchLabels:
      app: my-service
  template:
    metadata:
      labels:
        app: my-service
      annotations:
        prometheus: "true"
        prometheus/port: "8888"

You can use the following transformation:

- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: ${1}:${2}
    target_label: __address__

Source labels in prometheus are joined with ; and in this case we will have ie. (if your server by default listening on 8080): http://10.52.9.79:8080;8888

In regex we have:

• first group - all signs without :, ie.: //10.52.9.79
• second non-capturing group with original port if exist
• third group with port from annotation

As a result you set original address and port from annotation. Could be useful if for example you have Spring Boot application and you want to use different management port than default application port.