jmx_prometheus_javaagent configuration for Kafka Connect?

Question

The Prometheus jmx_exporter project gives an official example configuration for Kafka here:

https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-0-8-2.yml

Is there any similar configuration for Kafka Connect to export Kafka Connect JMX metrics to Prometheus?

Answer 1

The current solution that I'm using is the official Confluent Helm Charts and Docker image which has built-in Prometheus metric export:

https://github.com/confluentinc/cp-helm-charts/tree/master/charts/cp-kafka-connect

For reference, if you want a non-Docker, non-Kubernetes solution, a previous approach that works 100% is as follows:

1) Define this file kafka-connect-jmx.yaml. I am storing this config file in /usr/share/, you can put it elsewhere, but you will need to change the path reference below.

whitelistObjectNames: ["kafka.connect:*"]
lowercaseOutputName: false
lowercaseOutputLabelNames: false

rules:
  - pattern: 'status: running'
    value: 1
  - pattern: 'status: *'
    value: 0
  - pattern: ".*"

2) Download https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar. I put it in /usr/share/, you can put it elsewhere, but you will need to update the path in step 3.

3) When you run Kafka Connect connect-distributed make sure this environment variable is set and the paths match those used in steps 1+2

KAFKA_OPTS=-javaagent:/usr/share/jmx_prometheus_javaagent-0.12.0.jar=7072:/usr/share/kafka-connect-jmx.yaml