Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kafka Console consumer with kerberos authentication

How to consume published messages from the kafka (version 0.10) server which was kerberos authorized, for the authentication keytab file is being used.

I tried with the below command but no outputs were shown.

bin/kafka-console-consumer.sh --bootstrap-server :9092 --topic --from-beginning

like image 688
Raju Avatar asked Feb 12 '18 10:02

Raju


People also ask

How do you implement authentication in Kafka?

To enable authentication and authorization on the broker side, you need to perform two steps on each broker: Configure valid credentials. Configure the proper security protocol and authorizer implementation.

What is Keytab file in Kafka?

The useKeytab value is the full path to the Kerberos keytab file. The principal value is the Kerberos principal, for example user/host@REALM. Here, host is the host of the center for key distribution and REALM is the Kerberos REALM.

What is Kerberos in Kafka?

Kerberos Authentication. Kerberos is by far the most common option we see being used in the field to secure Kafka clusters. It enables users to use their corporate identities, stored in services like Active Directory, RedHat IPA, and FreeIPA, which simplifies identity management.


1 Answers

Kerberos-enabled clusters can pose some tricky challenges at times. I've had to deal with some of these myself.

If the Kafka Cluster is Kerberos-enabled then you'll need to supply a jaas.conf file with the Kerberos details. Try following these steps(they worked for me):

  1. Create a jaas.conf file with the following contents:
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="<path-to-the-keytab-file>"
principal="<kafka-principal>";
};

Note: I've assumed that the Kafka principal and the associated keytab is already created. If not, you'll need to create these first.

  1. Create a properties file (say "consumer.properties") with the following contents:
security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka
  1. Then at the terminal run the following command:
$export KAFKA_OPTS="-Djava.security.auth.login.config=<path-to-jaas.conf>"
  1. Execute the Kafka-console-consumer script:
$ kafka-console-consumer --topic <topic-name> --from-beginning 
--bootstrap-server <anybroker>:9092 --consumer.config <consumer.properties>

EDIT - Steps 3 and 4 could be combined just in case there is a preference to keep these as one command in the command history.

I hope this helps.

like image 103
Lalit Avatar answered Oct 12 '22 12:10

Lalit