How to define additional or custom tags for default Spring Boot 2 metrics?

Question

Recently I switched to Spring Boot 2 with Micrometer. As I got these shiny new metrics, I talked with our DevOps guys and we started exporting them to Telegraf.

To distinguish between different applications and application nodes, we decided to use tags. This works perfectly for custom metrics, but now I started thinking about the pre-defined. To achieve the same for default metrics, I need the ability to add extra tags for them as well.

Is it possible to achieve this? Am I doing this right?

EDIT: I tried next approach:

@Component
public class MyMetricsImpl implements MyMetrics {

    @Autowired
    protected MyProperties myProperties;
    @Autowired
    protected MeterRegistry meterRegistry;

    @PostConstruct
    public void initialize() {
        this.meterRegistry.config()
                .commonTags(commonTags());
    }

    @Override
    public List<Tag> commonTags() {
        List<Tag> tags = new ArrayList<>();
        tags.add(Tag.of("application", myProperties.getApplicationName()));
        tags.add(Tag.of("node", myProperties.getNodeName()));
        return tags;
    }
}

The problem is that my metrics behave correctly and even some of the Boot's metrics (at least http.server.requests) see my tags. But jvm.*, system.*, tomcat.* and many others still don't have the needed tags.

Answer 1

If you are looking for common tags support, you can do it by registering a MeterFilter doing it.

See this commit or this branch for an example.

With the upcoming Spring Boot 2.1.0.M1, you can use the following properties:

management.metrics.tags.*= # Common tags that are applied to every meter.

See the reference for details.

UPDATED:

As the question has been updated, I checked the updated question with this MeterFilter-based approach and confirmed it's working as follows:

Request: http://localhost:8080/actuator/metrics/jvm.gc.memory.allocated

Response:

{
  "name" : "jvm.gc.memory.allocated",
  "measurements" : [ {
    "statistic" : "COUNT",
    "value" : 1.98180864E8
  } ],
  "availableTags" : [ {
    "tag" : "stack",
    "values" : [ "prod" ]
  }, {
    "tag" : "region",
    "values" : [ "us-east-1" ]
  } ]
}

I didn't check the approach which has been provided in the updated question but I'd just use the proven MeterFilter-based approach unless there's any reason to stick with the approach.

2nd UPDATED:

I looked into the approach and was able to reproduce it with this branch.

It's too late to apply common tags in @PostConstruct as some metrics have been registered already. The reason why http.server.requests works is that it will be registered with the first request. Try to put a breakpoint on the point of filters' application if you're interested in it.

In short, try the above approach which is similar to the upcoming Spring Boot out-of-box support.

Answer 2

For Server:
(i.e. @RestController)
To add custom metrics tags for reactive-spring-boot application, in addition to default tags provided by the spring-boot framework.
provide one or more @Beans that implement WebFluxTagsContributor
OR
To replace the default tags, provide a @Bean that implements WebFluxTagsProvider.
Reference implementation which uses Spring-WebMVC as one of the answers.
Ref:

Following implementation is in groovy.

import io.micrometer.core.instrument.Tag
import org.springframework.boot.actuate.metrics.web.reactive.server.WebFluxTagsContributor
import org.springframework.http.HttpMethod
import org.springframework.http.HttpStatus
import org.springframework.stereotype.Component
import org.springframework.web.server.ServerWebExchange

@Component
class CustomWebClientExchangeTagsProvider implements WebFluxTagsContributor {

    final KEY = "key"

    /**
     * Provides tags to be associated with metrics for the given {@code exchange}.
     * @param exchange the exchange
     * @param ex the current exception (maybe {@code null})
     * @return tags to associate with metrics for the request and response exchange
     */
    @Override
    Iterable<Tag> httpRequestTags(ServerWebExchange exchange, Throwable ex) {
        String apiKey = exchange.request.queryParams[KEY] ?: "default_api_key"
        HttpStatus status = exchange.response.statusCode ?: HttpStatus.INTERNAL_SERVER_ERROR
        HttpMethod method = exchange.request.method ?: HttpMethod.OPTIONS
        String group = (status.value() >= 500 ? "5XX" : (status.value() >= 400) ? "4XX" : (status.value() >= 300) ? "3XX" : "NONE")

        Tag statusTag = Tag.of("status", status.value().toString())
        Tag methodTag = Tag.of("method", method.toString())
        Tag apiKeyTag = Tag.of(KEY, apiKey)
        Tag groupTag = Tag.of("group", group)

        return Arrays.asList(statusTag, methodTag, apiKeyTag, groupTag)
    }
}

Metrics tags:

Verify for client: metric http.client.requests.percentile or http.client.requests.

http://localhost:8010/metrics/http.client.requests

Verify for server:

http://localhost:8010/metrics/http.server.requests

similarly, tags will be added for http.server.requests.percentile metrics

{
    "name": "http.server.requests",
    "description": null,
    "base_unit": "milliseconds",
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 17.0
        },
        {
            "statistic": "TOTAL_TIME",
            "value": 8832.186054
        },
        {
            "statistic": "MAX",
            "value": 6.514132
        }
    ],
    "available_tags": [
        {
            "tag": "exception",
            "values": [
                "None",
                "ResponseStatusException"
            ]
        },
        {
            "tag": "method",
            "values": [
                "GET"
            ]
        },
        {
            "tag": "application",
            "values": [
                "myapplication"
            ]
        },
        {
            "tag": "uri",
            "values": [
                "/myapplication/v1/abcd",
                "/manage/metrics/{requiredMetricName}",
                "/manage",
                "/manage/metrics",
                "/myapplication/v1/windows",
                "/**"
            ]
        },
        {
            "tag": "outcome",
            "values": [
                "CLIENT_ERROR",
                "SERVER_ERROR",
                "SUCCESS"
            ]
        },
        {
            "tag": "key",
            "values": [
                "default_api_key",
                "[abcd]"
            ]
        },
        {
            "tag": "status",
            "values": [
                "404",
                "200",
                "502"
            ]
        },
        {
            "tag": "group",
            "values": [
                "4XX",
                "NONE",
                "5XX"
            ]
        }
    ]
}

OR

If you are using prometheus
you can verify custom tags as follows

http://localhost:8010/prometheus

Client:

http_client_requests_seconds_count{application="myapplication",clientName="myhostname.com",method="POST",outcome="CLIENT_ERROR",status="403",uri="/urlIamTryingToHit/v1/list",} 1.0

Not added for a client so there is no custom tags only existing tags provided by spring-boot.

Server:

http_server_requests_seconds_sum{application="myapplication",exception="None",group="4XX",key="default_api_key",method="GET",outcome="CLIENT_ERROR",status="404",uri="/manage/metrics/{requiredMetricName}",} 0.004154207

We can observe group="4XX", key="default_api_key", method="GET", status="404" along with existing default tags.

---

For Client:
To customize the tags, and depending on your choice of client, you can provide a @Bean that implements RestTemplateExchangeTagsProvider or WebClientExchangeTagsProvider.
There are convenience static functions in RestTemplateExchangeTags and WebClientExchangeTags.
Ref: