Don't print the kafka-console-consumer warnings

Question

I am trying to debug encrypted messages onto a Kafka cluster. Obviously these messages are full of non printable characters and are not usable on a console, so I wanted to save the output in a file like this:

kafka-console-consumer \
    --zookeeper 0.zookeeper.local,1.zookeeper.local \
    --max-messages 1 \
    --topic MYTOPIC > /tmp/message

I am unable to decrypt the resulting message, because the output contains, along with the ciphertext, warning messages such as:

[2016-02-24 11:52:47,488] WARN Reconnect due to socket error: null (kafka.consumer.SimpleConsumer)

Is there a way to get one single message in a file and NOT embed the warnings in that file?

Answer 1

There is a special logger config file for inbound console tools. Please, change the logger level from WARN to OFF in [kafka_home]/config/tools-log4j.properties file. The file should look like:

tools-log4j.properties

log4j.rootLogger=OFF, stdout 

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n

After it you will get only messages in console.

Answer 2

try create new config xml file and just run again

src/main/resources/logback.xml

<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>
    <logger name="org.apache.kafka" level="INFO"/>
    <logger name="org.apache.kafka.common.metrics" level="INFO"/>
    <root level="debug">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>