Expose kafka stream metrics with spring actuator (prometheus)

Question

I am running a Kafka Stream app with Springboot 2.

I would like to have my kafka stream metrics available in the prometheus format at host:8080/actuator/prometheus

I don't manage to have this. I am not sure I understand how kafka stream metrics are exported. Can actuator get these JMX metrics ? Is there a way to get these metrics and expose them in Prometheus format ?

PS: didn't worked with java jmx_prometheus_agent neither

Does someone has a solution or an example ?

Thank you

Answer 1

You could produce all available Kafka-Streams metrics (the same as from KafkaStreams.metrics()) into Prometheus using micrometer-core and spring-kafka libraries. For integrating Kafka-Streams with micrometer, you could have KafkaStreamsMicrometerListener bean:

@Bean 
KafkaStreamsMicrometerListener kafkaStreamsMicrometerListener(MeterRegistry meterRegistry) { 
    return new KafkaStreamsMicrometerListener(meterRegistry); 
}

where MeterRegistry is from micrometer-core dependency.

If you create Kafka Streams using StreamsBuilderFactoryBean from spring-kafka, then you need to add listener into it:

streamsBuilderFactoryBean.addListener(kafkaStreamsMicrometerListener);

And if you create KafkaStreams objects directly, then on each KafkaStreams object you need to invoke

kafkaStreamsMicrometerListener.streamsAdded(beanId, kafkaStreams);

where beanId is any unique identifier per KafkaStreams object.

As a result, Kafka Streams provides multiple useful Prometheus metrics, like kafka_consumer_coordinator_rebalance_latency_avg, kafka_stream_thread_task_closed_rate, etc. KafkaStreamsMicrometerListener under the hood uses KafkaStreamsMetrics.

If you need to have Grafana Prometheus graphs with these metrics, you need to add them as Gauge metric type.