Kafka on kubernetes cluster with Istio

Question

I have k8s cluster with Istio v1.6.4. The sidecar injection is disabled by default. I have Kafka cluster running on this k8s installed with strimzi kafka operator. The Kafka cluster works without any problems when kafka as well as client pods doesn't have Istio-proxy injected. My problem: When I create a pod with kafka client and Istio-proxy injected I can't connect to Kafka cluster. The logs on client side:

java.io.IOException: Connection reset by peer

and on the server side: org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 369295616 larger than 104857600)

After some googling and checking the Istio-proxy logs it turns out the problem is that Istio-proxy connects to kafka plaintext endpoint with TLS. I can workaround this by setting the default PeerAuthentication with mtls.mode: DISABLED but I don't want to set global setting for it.

What is strange if I create a simple k8s service and run the netcat "server" on pod running kafka server and netcat "client" on pod running kafka client - everything works fine.

I have 2 question:

1. Why the kafka Istio-proxy behaves different when connecting to Kafka cluster than other TCP connections (like using nc)?
2. How to disable mtls for one host only? I was playing with PeerAuthentication but no luck...

Answer 1

With jt97's help I was able to solve this problem.

As I wrote I'm using Strimzi Operator to install kafka cluster on k8s. It creates 2 services:

• kafka-bootstrap - which is a regular service with ClusterIP
• kafka-brokers - a headless service.

In my case the full name of the services are kafka-kafka-operated-kafka-bootstrap and kafka-kafka-operated-kafka-brokers, respectively.

I created a DestinationRule:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: kafka-no-tls
spec:
  host: "kafka-kafka-operated-kafka-brokers"
  trafficPolicy:
    tls:
      mode: DISABLE

and used the headless service when connecting to kafka:

kafka-topics --bootstrap-server kafka-kafka-operated-kafka-brokers:9092 --list
__consumer_offsets
_schemas

and it worked as expected.

BTW, setting tls.mode to SIMPLE didn't help.

To be honest I still don't understand why in this particular case Istio-proxy by default (without DestinationRule) tries to connect with TLS - according to documentation:

By default, Istio tracks the server workloads migrated to Istio proxies, and configures client proxies to send mutual TLS traffic to those workloads automatically, and to send plain text traffic to workloads without sidecars.