Kafka + using prometheus have confilcts with kafka cli commands

Question

We are using Prometheus for some time and really enjoying it.

Few words about What is jmx-exporter

jmx-exporter is a program that reads JMX data from JVM based applications (e.g. Java and Scala) and exposes it via HTTP in a simple text format that Prometheus understand and can scrape.

So let’s get started with my issue …

We configured the kafka with jmx exporter as the following

export KAFKA_OPTS="-javaagent:/home/jmx_prometheus_javaagent-0.11.0.jar=7071:/home/kafka-2_0_0.yml"

this configuration set in ambari under kakfa config

after setting the configuration we restart all 3 kafka brokers

we Check that jmx-exporter HTTP server is listening:

netstat -tlnp | grep 7071
tcp6       0      0 :::7071                 :::*                    LISTEN      63872/java

And scrape the metrics!

curl -s localhost:7071 | grep -i kafka | head
# HELP kafka_log_logcleanermanager_max_dirty_percent Attribute exposed for management (kafka.log<type=LogCleanerManager, name=max-dirty-percent><>Value)
# TYPE kafka_log_logcleanermanager_max_dirty_percent gauge
kafka_log_logcleanermanager_max_dirty_percent 0.0

until now every thing is cool

but when we start to use kafka commands , for example to print the list of topics we get:

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh –zookeeper $zookeeper_server:2181 –list

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
        at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at sun.net.httpserver.ServerImpl.bind(ServerImpl.java:133)
        at sun.net.httpserver.HttpServerImpl.bind(HttpServerImpl.java:54)
        at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer.<init>(HTTPServer.java:145)
        at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:49)
        ... 6 more
FATAL ERROR in native method: processing of -javaagent failed

Note – before we add the following line:

export KAFKA_OPTS="-javaagent:/home/jmx_prometheus_javaagent-0.11.0.jar=7071:/home/kafka-2_0_0.yml "

everything was ok

so any advice – how to start to solve this problem from this point ?

more reference -

https://alex.dzyoba.com/blog/jmx-exporter

https://medium.com/@mousavi310/monitor-apache-kafka-using-grafana-and-prometheus-873c7a0005e2

Answer 1

Of all the solutions I found, it helped me to change the KAFKA_OPTS variable to EXTRA_ARGS. But I still don't understand the cause of the conflict. Kafka runs on one port and javaagent on another. But nevertheless, when creating topics, an error appears. Javaagent I am using to pass Prometheus metrics