How to deploy Kafka Streaming Application on Kafka Cluster

Question

I wrote the Kafka Streaming application and I want to deploy it on Kafka cluster. So I built a jar file and run it using the command:

 java -jar KafkaProcessing-1.0-SNAPSHOT-jar-with-dependencies.jar testTopic kafka1:9092,kafka2:9092 zookeeper1:2181,zookeeper2:2181 output

It runs correctly but the job is running on the machine I run above command! I thought when I specify BOOTSTRAP-SERVERS it automatically does computing on cluster, not on the host machine!

So my question is how can I submit Kafka streaming job on kafka cluster? Like Spark and Flink that provided commands spark-submit and flink run to deploy applications on the cluster.

Answer 1

Kafka streams has different architecture - it doesn't need cluster orchestration like Spark/Flink - they are just normal applications that you can start and stop, and if you start - they will be scaled up, if you stop they scale down. Internally they are using Kafka to coordinate data processing, similarly to other Kafka consumers.

If you have Kubernetes, Docker Swarm, or other similar platform, then you can pack your app into Docker, and use that platform to run your Kafka Streams app.

Answer 2

At my organization we are using the kafka streams application. We had explored this option of deploying on the server. This facility is simply not provided. You only have the option of running kafka streams application wherever you are running it. There exists no job submission option yet.