I have a problem communicating with Kafka secured with sasl
using console scripts. Kafka is secured with sasl
, listener is SASL_PLAINTEXT
and mechanism is PLAIN
.
What I did: I tried listing some data using one of kafka scripts:
bin/kafka-consumer-groups.sh --bootstrap-server (address) --list
However I get
WARN Bootstrap broker (address) disconnected (org.apache.kafka.clients.NetworkClient)
and command fails, which is understandable because it's secured with sasl.
So I tried how to add client username/password to that command.
First, I tried to run kafka-console-consumer
script, I used --command-config
to add necessary file. I quickly discovered that I can't add jaas
file directly and I needed to use .properties
file, so I did.
My properties file(keep in mind that brackets indicate "censored" data, I can't put all real data here):
bootstrap.servers=(address)
zookeeper.connect=127.0.0.1:2181
zookeeper.connection.timeout.ms=6000
sasl.jaas.config=(path)/consumer_jaas.conf
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
group.id=(group)
My jaas file:
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username=(username)
password=(password);
};
This jaas
file works in my standard java applications.
However, when I'm trying to run either kafka-consumer-groups
script or kafka-console-consumer
, I get this error:
Exception in thread "main" org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: Login module not specified in JAAS config
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:94)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:93)
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:51)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:84)
at kafka.admin.AdminClient$.create(AdminClient.scala:229)
at kafka.admin.AdminClient$.create(AdminClient.scala:223)
at kafka.admin.AdminClient$.create(AdminClient.scala:221)
at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.createAdminClient(ConsumerGroupCommand.scala:454)
at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.<init>(ConsumerGroupCommand.scala:389)
at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:65)
at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
Caused by: java.lang.IllegalArgumentException: Login module not specified in JAAS config
at org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:68)
at org.apache.kafka.common.security.JaasUtils.jaasConfig(JaasUtils.java:59)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:85)
This jaas
file is a direct copy of a file that I'm using in java app that communicates with kafka and it works, however here, using console tools, it just doesn't work. I tried searching for a solution but I can't find anything useful.
Can anyone help me with this?
The Java Authentication and Authorization Service (JAAS) API supplies user authentication and authorization services for Java applications. After enabling Kerberos, Ambari sets up a JAAS login configuration file for the Kafka server.
SASL/PLAIN Overview. PLAIN, or SASL/PLAIN, is a simple username/password authentication mechanism that is typically used with TLS for encryption to implement secure authentication.
Via the Java property: java.security.auth.login.config. In this case, you set it to the path of your JAAS file. Also if you set it in KAFKA_OPTS, kafka-consumer-groups.sh will pick it up automatically. I put jaas configuration entry directly into .properties file and it worked. Now I can use these scripts without any errors.
Kafka uses the Java Authentication and Authorization Service ( JAAS ) for SASL configuration. You must provide JAAS configurations for all SASL authentication mechanisms. There are two ways to configure Kafka clients to provide the necessary information for JAAS:
The Login Configuration File for the JAAS Authentication Tutorial describes sample_jaas.config, which is a sample login configuration file used by both tutorials. sampleacn.policy is a sample policy file granting permissions required by the code for the authentication tutorial.
As you've already figured out, you can use --command-config to pass a properties file to kafka-consumer-groups.sh. Via the Java property: java.security.auth.login.config. In this case, you set it to the path of your JAAS file. Also if you set it in KAFKA_OPTS, kafka-consumer-groups.sh will pick it up automatically.
There are 2 ways to provide the JAAS configuration to the Kafka clients.
Via the client property: sasl.jaas.config
. In that case you set it to the actual JAAS configuration entry. For example, your configuration file becomes:
bootstrap.servers=(address)
zookeeper.connect=127.0.0.1:2181
zookeeper.connection.timeout.ms=6000
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="(username)" password="(password)";
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
group.id=(group)
As you've already figured out, you can use --command-config
to pass a properties file to kafka-consumer-groups.sh
.
Via the Java property: java.security.auth.login.config
. In this case, you set it to the path of your JAAS file. Also if you set it in KAFKA_OPTS
, kafka-consumer-groups.sh
will pick it up automatically.
export KAFKA_OPTS="-Djava.security.auth.login.config=(path)/consumer_jaas.conf"
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With