Kafka Streams 2.5.0 requires input topic

Question

Starting with Kafka Streams 2.5.0 it seems like a topology must include an input topic. In Kafka 2.4.1 (and earlier) that is not the case.

I have an application where the topology is just creating a few global state stores that read in data from topics written to by other applications.

With Kafka 2.5.0 I get this error:

13:24:27.161 [<redacted>-7cf1b5c9-4a6e-4bf2-9f77-f7f85f2df3bb-StreamThread-1] ERROR o.a.k.s.p.internals.StreamThread - stream-thread [<redacted>-7cf1b5c9-4a6e-4bf2-9f77-f7f85f2df3bb-StreamThread-1] Encountered the following error during processing:
java.lang.IllegalStateException: Consumer is not subscribed to any topics or assigned any partitions
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1228)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
    at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:853)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:753)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:697)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:670)

If I add a dummy input topic (e.g. via streamsBuilder.stream(Pattern.compile("hack"));) the application starts fine.

Is this behavior to be expected or is it an unintentional change in Kafka Streams 2.5.0?

More details: The use case above may seem a bit weird and I would have to agree. The reason for doing it in the first place was a shortcoming of Interactive Queries where for periods of time the application could not answer queries. I see that issue has been fixed in Kafka Streans 2.5.0 via KIP-535 which is great. I hope to look into IQ again later.

Answer 1

If you don't have any non-global parts of your topology, there's no reason to have any StreamThreads at all. Meaning you can easily work around this by just setting num.threads to zero -- arguably, you should be doing this anyways to avoid unnecessary overhead and group coordination. Setting this to zero by default when a global-only topology is detected is the "fix" anyway, so you don't need to wait for that