Stop a Kafka Streams app

Question

Is it possible to have a Kafka Streams app that runs through all the data in a topic and then exits?

Example I'm producing data into topics based on date. The consumer gets kicked off by cron, runs through all the available data and then .. does what? I don't want it to sit and wait for more data. Just assume it's all there and then exit gracefully.

Possible?

Answer 1

In Kafka Streams (as for other stream processing solutions), the is no "end of data" because it is stream processing in the first place -- and not batch processing.

Nevertheless, you could watch the "lag" of your Kafka Streams application and shut it down if there is not lag (lag, is the number of not yet consumed messages).

For example, you can use bin/kafka-consumer-groups.sh to check the lag of your Streams application (the application ID is used as consumer group ID). If you want to embed this in your Streams applications, you can use kafka.admin.AdminClient to get consumer group information.

Answer 2

You can create a consumer and then once it stops pulling up data, you can have call consumer.close(). Or if you want to poll again in the future just call consumer.pause() and call .resume later.

One way to do this is within the consumer poll block. Such as

data = consumer.poll()
if (!data.next()) {
   consumer.close()
}

Keep in mind that poll returns ConsumerRecord<K,V> and conforms to the Iterable interface.