Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Implement Kafka Streams Processor in .Net?

Is that possible?

The official .Net client confluent-kafka-dotnet only seems to provide consumer and producer functionality.

And (from what I remember looking into Kafka streams quite a while back) I believe Kafka Streams processors always run on the JVMs that run Kafka itself. In that case, it would be principally impossible.

like image 561
Evgeniy Berezovsky Avatar asked Feb 16 '18 02:02

Evgeniy Berezovsky


People also ask

Can Kafka be used for stream processing?

Kafka Streams is a client library for processing and analyzing data stored in Kafka. It builds upon important stream processing concepts such as properly distinguishing between event time and processing time, windowing support, and simple yet efficient management and real-time querying of application state.

Can we use Kafka in C#?

To make your C# code understand how to produce and consume messages, you need a client for Kafka. The most used client today is Confluent's Kafka . NET Client. To install it, right-click the solution and select the Manage NuGet Packages for Solution…

What is stream processor in Kafka?

A stream processor is a node in the processor topology that represents a single processing step. With the Processor API, you can define arbitrary stream processors that processes one received record at a time, and connect these processors with their associated state stores to compose the processor topology.


1 Answers

Yes, it is possible to re-implement Apache Kafka's Streams client library (a Java library) in .NET. But at the moment there doesn't exist such a ready-to-use Kafka Streams implementation for .NET.

And (from what I remember looking into Kafka streams quite a while back) I believe Kafka Streams processors always run on the JVMs that run Kafka itself. In that case, it would be principally impossible.

No, Kafka Streams "processors" as you call them do not run in (the JVMs of) the Kafka brokers, which would be server-side.

Instead, the Kafka Streams client library is used to implement client-side Java/Scala/Clojure/... applications for stream processing. These applications talk to the Kafka brokers (which form the Kafka cluster) over the network.

like image 92
Michael G. Noll Avatar answered Sep 22 '22 00:09

Michael G. Noll