Kafka Consumer metrics gone upon upgrade from Spring Boot 2.2.2 to 2.3.0

Question

Problem:

We upgraded our Spring Boot version from 2.2.2 to 2.3.0, and all of the kafka_consumer_* metrics that were seen in the Prometheus endpoint in 2.2.2 are not visible in 2.3.0.

For example, all of the below are missing:

• kafka_consumer_records_consumed_total_records_total
• kafka_consumer_records_lag_records
• kafka_consumer_fetch_latency_max_seconds
• kafka_consumer_bytes_consumed_total_bytes_total

Not sure if we're missing some kind of configuration or something buried in the docs...

What has been tried:

• Combed the Spring Boot 2.3.0 release notes, updated micrometer documentation, and updated spring-kafka documentation for why this might be happening
• Googled to what feels like the ends of the Earth
• Tried upgrading to Spring Boot 2.2.7 and the kafka metrics are still there, only upgrading to 2.3.0 seems to cause the issue
• Removed any non-needed dependency/customization that was in our code for the project, and bare bones just connected to a kafka container on the localhost, and the metrics still don't appear

Relevant code/details:

• We're using Red Hat AMQ Streams for our kafka broker (kafka version 2.1.1)
• The only thing we've changed in our environment is the Spring Boot version (and dependencies that are automatically pulled in/updated) to recreate this issue

Below is our build.gradle.kts before the change:

import org.jetbrains.kotlin.gradle.tasks.KotlinCompile

plugins {
    id("org.springframework.boot") version "2.2.2.RELEASE"
    id("io.spring.dependency-management") version "1.0.9.RELEASE"
    kotlin("jvm") version "1.3.72"
    kotlin("plugin.spring") version "1.3.72"
}

group = "ourGroup"
version = "0.0.1"
java.sourceCompatibility = JavaVersion.VERSION_1_8

repositories {
    mavenCentral()
}

extra["springCloudVersion"] = "Hoxton.RELEASE"

dependencyManagement {
    imports {
        mavenBom("org.springframework.cloud:spring-cloud-dependencies:${property("springCloudVersion")}")
    }
}

dependencies {
    implementation("org.springframework.cloud:spring-cloud-starter-stream-kafka")
    implementation("org.springframework.boot:spring-boot-starter-actuator")
    implementation("io.micrometer:micrometer-registry-prometheus")
    implementation("org.springframework.boot:spring-boot-starter-webflux")
    implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
    testImplementation("org.springframework.boot:spring-boot-starter-test") {
        exclude(group = "org.junit.vintage", module = "junit-vintage-engine")
    }
    testImplementation("io.projectreactor:reactor-test")
    testImplementation("org.springframework.security:spring-security-test")
}

tasks.withType<Test> {
    useJUnitPlatform()
}

tasks.withType<KotlinCompile> {
    kotlinOptions {
        freeCompilerArgs = listOf("-Xjsr305=strict")
        jvmTarget = "1.8"
    }
}

Now, if we just update our build.gradle.kts with a new spring boot version like in the below line our kafka metrics disappear:

    id("org.springframework.boot") version "2.3.0.RELEASE"

Below are screenshots of the prometheus metrics we were seeing before and after the change:

Before 2.3.0 upgrade

After 2.3.0 upgrade

Thanks in advance for the help! Let me know if you need any other details!

Answer 1

It works for normal spring-kafka consumers

@SpringBootApplication
public class So62292889Application {

    public static void main(String[] args) {
        SpringApplication.run(So62292889Application.class, args);
    }

    @KafkaListener(id = "so62292889", topics = "so62292889")
    public void listen(String in) {
        System.out.println(in);
    }

    @Bean
    public NewTopic topic() {
        return TopicBuilder.name("so62292889").partitions(1).replicas(1).build();
    }

    @Bean
    public ApplicationRunner runner(MeterRegistry registry) {
        return args -> {
            registry.getMeters().forEach(meter -> System.out.println(meter.getId()));
        };
    }

}

MeterId{name='kafka.consumer.outgoing.byte.total'...

I see you are using spring-cloud-stream. The problem is that this project creates its own producer/consumer factories and doesn't add the Micrometer listeners.

I opened an issue against the binder.

Answer 2

Similar to @jumping_monkey, I have a spring-kafka app using Spring Boot 2.3.0.RELEASE, and the kafka_consumer_* metrics were not showing up in /actuator/prometheus. They did show up when I downgraded to Spring Boot 2.2.7.RELEASE.

It has been determined that an issue exists for apps using spring-cloud-stream, and there is a plan to fix that. My app does not use spring-cloud-stream, however; it uses spring-kafka.

The solution was to follow the guidance on the spring-kafka reference to add a MicrometerConsumerListener to the consumer factory. After doing this, metrics show up as expected.