AWS Kafka (MSK) - How to generate Keystore and truststore and use the same in my Spring Cloud Stream application?

Question

Is there any info as to how can I use the AWS MSK details in my Spring Cloud Stream application ?

I believe we need to generate a keystore and truststore and then incorporate the same in our application ? I went through the "Client Authentication" page of the AWS MSK and found that to be very confusing.

Can anyone help me with steps on this ? I am just trying to deploy this application which uses the AWS MSK (3 brokers).

Thank you.

Answer 1

Short answer: Your kafka client will need this in the configuration:

# security settings
security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks
ssl.truststore.password=
ssl.endpoint.identification.algorithm=

That is if you use the same JVM truststore from the tutorial, and no password. The ssl.endpoint.identification.algorithm turns off the host name verification.

Long answer: I wondered the same thing after going through the tutorial, wondering why the JVM truststore magically works when connecting to MSK. The explanation is this:

If you take a peek at what certificates this truststore imported

keytool --list -v -keystore /tmp/kafka.client.truststore.jks | grep Owner

One of them is Starfield Services Root Certificate Authority, when Amazon purchased the company, the CA became one of Amazon's (see all of them here https://www.amazontrust.com/repository/). Since JVM truststore trusts this CA, it also trusts anything signed by the CA, and the MSK cluster is one of them.

If you would prefer to generate your own truststore, download one of the Amazon's certificate and import

keytool -keystore kafka.client.truststore.jks -alias CARoot -importcert -file {downloaded-cert} -storepass {your-password}

Thanks, Yanan