Kafka Streams app does NOT fail when the Kafka cluster goes down

Question

I have a Kafka Streams app running (0.10.2.1). When I shut down the Kafka cluster the streams app continues to wait for the next message, when the cluster is brought back up, it will resume consuming messages. For the duration that the cluster is down the app appears to be working fine. I have tested this for over 45 minutes.

I would expect Kafka to throw an exception or stop. I have configure a StateListener to log when KafkaStreams shuts down, however it is never invoked.

kafkaStreams.setStateListener((newState, _) => {
  if (newState == KafkaStreams.State.NOT_RUNNING) {
    Log.error("Kafka died unexpectedly.")
  }
})

How do I get Kafka to throw an exception or shutdown when it cannot connect to the cluster?

Note: this assumes that cluster goes down after the app has started

Answer 1

Why would you want the Kafka Streams app to go down?

The app should be resilient to broker failures, that is, keep going patiently until the broker recovers and it seems that this is what it's doing. If you have multiple instances of the Kafka Streams application and one of them loses connectivity to the broker, the load will be re-balanced onto the remaining instances. If each instance that lost connectivity just shut itself down, you would be losing instances and with them losing redundancy and parallelism even if the broker connectivity recovered. The way it is now Kafka Streams is designed for resilience. I'd argue that this is the correct behaviour.

IMHO if you want to detect broker (or connectivity) failures, that's a use case for monitoring, not for introducing failures into Kafka Streams applications.